PAANIC

The setup

Robotic advancements call for the further application of Natural Language Processing (NLP) and the ability of robots to understand natural language instructions from humans. As such, we have developed a method to reduce mistakes in robotic following of human commands by implementing hazard tagging on natural language instructions, which can then be modified into the real-life application of robotic grip strength, general procedure for a task, and navigation of dangerous situations.

In this project, we trained 4 neural network models on the task of assigning hazard tags [Fragile, Hot, Sharp, Slippery, Electricity, Water] to a variety of household task instructions from the TEACh dataset. We personally tagged our data, and chose to utilize a recurrent neural network (rnn), a rnn passing results to a feed-forward network, a rnn with long short-term memory units (LSTMs), and a pre-trained BERT model passing results to a feed-forward network.

Overall, non-recurrent models perform the worst on this task. Classical recurrent models perform better than non-recurrent and models with attention, like BERT, perform the best. These results are not particularly surprising as transformer’s outperform other models on most natural language processing tasks. However, an interesting result to note is that the basic RNN model outperformed the LSTM model. LSTM architecture was originally conceived as a way to extend the interaction distance between tokens in a recurrent network. However, most relevant interactions for this task are fairly immediate and usually prepositional. Models had trained until epochs no longer increased accuracy performance, and we found that the better-performing models tended to require fewer epochs to train.

As an output, we produced a dataset of TEACh Dataset instructions with corresponding hazard tagging. This research was done in conjunction with, and in part for, Northeastern University's research presentation event, RISE. This work is supported with a powerpoint presentation and a paper on our research.

Reflections

An immediate next step would be to improve performance by expanding the dataset and adding more variety to the types of instructions. It could also be incredibly useful to further research which tags to use to be most effective- I'd be interested in using a 'liquid' hazard tag, given that context or programming could provide 'wet' or 'slippery'.

On that note, it would be rewarding to look at integration with a robot- such as with a multimodal approach or looking at the robotic controller.