Assistive Recipe Editing through Critiquing
Diego Antognini, Shuyang Li, Boi Faltings, Julian McAuley
2023 EACL
pdf

We present RecipeCrit, a hierarchical denoising auto-encoder that edits recipes given ingredient-level critiques. The model is trained for recipe completion to learn semantic relationships within recipes. Our work's main innovation is our unsupervised critiquing module that allows users to edit recipes by interacting with the predicted ingredients; the system iteratively rewrites recipes to satisfy users' feedback. Experiments on the Recipe1M recipe dataset show that our model can more effectively edit recipes compared to strong language-modeling baselines, creating recipes that satisfy user constraints and are more correct, serendipitous, coherent, and relevant as measured by human judges.

SHARE: a System for Hierarchical Assistive Recipe Editing
Shuyang Li, Yufei Li, Jianmo Ni, Julian McAuley
2022 EMNLP
pdf | video | data

We introduce SHARE: a System for Hierarchical Assistive Recipe Editing to assist home cooks with dietary restrictions---a population under-served by existing cooking resources. Our hierarchical recipe editor makes necessary substitutions to a recipe's ingredients list and re-writes the directions to make use of the new ingredients. We show that recipe editing is a challenging task that cannot be adequately solved with human-written ingredient substitution rules or straightforward adaptation of state-of-the-art models for recipe generation.

Self-Supervised Bot Play for Conversational Recommendation with Rationales
Shuyang Li, Bodhisattwa Prasad Majumder, Julian McAuley
2022 RecSys
pdf | acm | q&a | hub

We introduce a framework for training interactive recommender systems for multi-turn conversational recommendation using self-supervised bot-play predicated on review data. We demonstrate via experiments on three real-world datasets that our method is model-agnostic and allows simple matrix factorization and linear recommender systems to out-perform state-of-the-art existing techniques for conversational recommendation.

Instilling type knowledge in language models via multi-task QA
Shuyang Li, Mukund Sridhar, Chandana Satya Prakash, Jin Cao, Wael Hamza, Julian McAuley
2022 NAACL (Findings)
pdf | data

We introduce a novel multi-task QA framework for pre-training language models on joint articles and knowledge bases to recognize types, entities, and their relationship in a type ontology. We demonstrate that generative language models trained using our framework can a) achieve up to 14.9% (absolute) / 49.4% (relative) improvement in zero-shot dialog state tracking (DST) over current state-of-the-art, b) infer types for previously unseen entities better than other pre-trained language models (+16.7 F1), and c) discover novel complex and meaningful types. We also release the WikiWiki dataset linking 10M Wikipedia articles with 41K unique types from the Wikidata knowledge graph.

Zero-shot Generalization in Dialog State Tracking through Generative Question Answering
Shuyang Li, Jin Cao, Mukund Harakere Sridhar, Henry Zhu, Daniel Li, Wael Hamza, Julian McAuley
2021 EACL
pdf

We introduce a novel ontology-free framework that supports natural language queries for unseen constraints and slots in multi-domain task-oriented dialogs. Our approach is based on generative question-answering using a conditional language model pre-trained on substantive English sentences. Our model improves joint goal accuracy in zero-shot domain adaptation settings by up to 9\% (absolute) over the previous state-of-the-art on the MultiWOZ 2.1 dataset.

Interview: Large-scale Modeling of Media Dialog with Discourse Patterns and Knowledge Grounding
Bodhisattwa P. Majumder*, Shuyang Li*, Jianmo Ni, Julian McAuley
2020 EMNLP
pdf | data | code

We perform the first large-scale analysis of discourse in media dialog and its impact on generative modeling of dialog turns, with a focus on interrogative patterns and use of external knowledge. We introduce Interview—a large-scale (105K conversations) media dialog dataset collected from news interview transcripts—which allows us to investigate such patterns at scale.

Speech Recognition and Multi-Speaker Diarization of Long Conversations
Henry Huanru Mao, Shuyang Li, Julian McAuley, Gary W. Cottrell
2020 INTERSPEECH
pdf | data

We compare separate and joint frameworks for speech recognition (ASR) and speaker diarization in a multi-speaker setting, showing that joint models can perform well in settings without known utterance bounds. We release a dataset to support multi-speaker ASR and diarization, drawn from a weekly radio program: This American Life.

Interview: A Large-Scale Open-Source Corpus of Media Dialog
Bodhisattwa P. Majumder*, Shuyang Li*, Jianmo Ni, Julian McAuley
2020 ArXiv preprint (CoRR)
pdf | data

We introduce a large-scale dataset of media dialog: interview-style transcripts from NPR talk shows. We demonstrate its usefulness for dialog modeling and its special traits and dialog structure when compared to existing small-scale spontaneous dialog transcriptions and written proxies for conversation.

Generating Personalized Recipes from Historical User Preferences
Bodhisattwa P. Majumder*, Shuyang Li*, Jianmo Ni, Julian McAuley
2019 EMNLP
pdf | code | data | news 1 2 3

We propose the task of personalized recipe generation: expanding a name and incomplete ingredient details into a complete natural-text recipe instruction set, aligned with a user's historical preferences; we present a model that attends on user activity traces to solve this task.

Expression of the transient receptor potential channels TRPV1, TRPA1 and TRPM8 in mouse trigeminal primary afferent neurons innervating the dura
Dongyue Huang, Shuyang Li, Ajay Dhaka, Gina M Story, Yu-Qing Cao
Molecular Pain 2012, 8:66
pdf

We study the size and clustering of TRP-channel expressing neurons in the trigeminal dura. Our results suggest that TRPV1 and TRPA1 but not TRPM8 channels likely contribute to the excitation of dural afferent neurons and the subsequent activation of the headache circuit. These results provide an anatomical basis for understanding further the functional significance of TRP channels in headache pathophysiology.

Variable Bitrate Discrete Neural Representations via Causal Self-Attention
Shuyang Li*, Huanru Henry Mao*, Julian McAuley
2021 NeurIPS Workshop on Pre-registration in Science
pdf | poster | video

Generative modeling across text, video, and image domains has benefited from the introduction of discrete (quantized) representations for continuous data. We aim to learn a single model able to produce discrete representations at different granularities (bitrates). We propose a framework based on the Perceiver IO architecture, incorporating causal attention to learn ordered latent codes that can then be adaptively pruned to a target compression rate.

Bernard: A Stateful Neural Open-domain Socialbot
Bodhisattwa Prasad Majumder, Shuyang Li, Jianmo Ni, Huanru Henry Mao, Sophia Sun, Julian McAuley
Alexa Prize Grand Challenge 3 Proceedings (2019)
pdf | press

We propose Bernard: a framework for an engaging open-domain socialbot. We explore various strategies to generate coherent dialog given an arbitrary dialog history. We incorporate a stateful autonomous dialog manager using non-deterministic finite automata to control multi-turn conversations.

Recipes for Success: Data Science in the Home Kitchen
Shuyang Li, Julian McAuley
Harvard Data Science Review
pdf | link

We survey the history of data science and machine learning methods to assist home cooks find and make nutritious, delicious meals. This article explores domains including recipe recommendation, recipe generation, image-to-recipe retrieval, and assistive technologies that integrate with the cooking process.

Exploring Rich Features for Sentiment Analysis with Various Machine Learning Models
Shuyang Li, Xiaoyan Li
IEEE Undergraduate Research Technology Conference 2016, Princeton Senior Thesis 2016
pdf | poster | data

This was my senior thesis at Princeton University, supervised by Dr. Xiaoyan Li. Here, we investigate the use of rich features to extend the bag-of-words model for sentiment analysis using machine learning in the movie review domain. We focus on subjectivity analysis and sentence position features. In addition, I created a manually labeled set of subjective and summary sentences for 2000 reviews in the Cornell IMDB movie review corpus.

Learning Rate Analysis for Temporal-Difference Learning
Work in Castle Labs, supervised by Warren Powell
Summer 2014
pdf | framework

The objective was to find optimal policies in an energy allocation problem. The model contained a battery, power grid, wind energy, and demand, with the latter three stochastic variables. I investigated the performance of Q-Learning and SARSA given different learning rates.

Meta: Natural Language Processing
Research Scientist

Heterogeneous Knowledge Graphs, Text Generation (August 2022 - Present)

Salesforce AI: Natural Language Processing
Research Scientist Intern

Meta-Learning for Low-Resource Dialog Agents (October 2021 - February 2022)

Amazon Alexa: Alexa Natural Language Understanding
Applied Scientist Intern

Concept-centric Pre-training for Language Modeling (June 2021 - October 2021)
At Alexa NLU, I researched ways to instill knowledge about entities and types in large language models. I investigated how such pre-training can be posed as question-answering and how such models enable strong entity typing and generalization performance in dialog state tracking.

Generalizable Dialog State Tracking (June 2020 - October 2020)
At Alexa NLU, I conducted research on generative approaches toward generalizable dialog state tracking of user preferences in a domain adaptation setting. I investigated knowledge transfer in autoregressive language models for question-answering based dialog understanding. I also analyzed common error modalities in preference and belief tracking for task-oriented dialog.

Google Cloud AI: Kaggle
Software Engineering Intern

Dataset Categorization and Search (June 2019 - September 2019)
At Kaggle, I built a framework for automatically generating semantic tags for datasets based on free-text metadata. I also implemented metrics for dataset discoverability and search success. By the end of the summer, I had more than doubled the size of our tag ontology and tripled tag coverage across all public datasets on Kaggle.

Bloomberg: Structured Products
Senior Software Engineer

Data Quality Control Platform (June 2017 - September 2018)
I helped to design and implement a Spark-based infrastructure for high bandwidth data processing jobs. We wrote a series of applications to wrap common data-access, filtering, and testing paradigms, so other members of the Structured Products group could write test logic as a self-contained plugin and farm the job out to our Spark cluster. We achieved a 100x+ speedup for some jobs due to parallelization, and abstracted away Spark application boilerplate.

Semi-Structured Text Clustering for Securities (June 2017 - September 2018)
I investigated how various mortgage-backed securities (pools, generics, CMOs) could be clustered by the shape of the data we regularly received for each security. This entailed identifying data shape features for semi-structured text data received for over 2.5 million securities and clustering on time-series data for each security.

Goldman Sachs: Operations Analytics Strats
Technology Analyst

Machine Learning Platform as a Service (July 2016 - June 2017)
We built a platform for data science and machine learning on top of a centralized data store, to allow anybody to classify and regress on arbitrary datasets without needing a deep background in programming for machine learning. One client was an internal data platforms team, which used our platform to better predict expected runtimes for their data ingestion workflow.

FIX Market Data Pipeline (July 2016 - August 2017)
This was the project team I worked on for Grey Wolf on Goldman Sachs Asset Management - Fixed Income (GSAM FI). Over 6 weeks, we built a system to consume FIX messages from several marketplaces, parse them, and store the messages in a database. Our application allowed the GSAM FI team to consume a vastly greater quantity of market messages and made it available for further analysis by traders. It is currently a production system.

Goldman Sachs: Operations Analytics Strats
Summer Analyst

Automated Invoice Recognition and Extraction (June 2015 - August 2015)
We worked with Accounting Services to build a tool that could automatically index certain values from images of invoices. They received invoices through email or fax and scanned them into .tiff files. Our algorithm relied on segmentation of individual values to produce structured output and templating to identify likely segment locations.

Workflow Assignment through Mixed Integer Linear Programming (June 2015 - August 2015)
We worked with several different Operations groups to produce a tool for automatically assigning tasks to available analysts. This took into account available analysts, their expected bandwidth for the remainder of the day, their proficiency with incoming tasks, and implemented a Maker-Checker process. The tool is to be run by managers, and outputs a list of each person's assignments as well as a list of unassigned tasks with reasoning listed.