Controller for the Dobot Magician with Nextion Touchscreen

A few months ago I made an Arduino based controller, which allows a  Dobot Magician robot arm to be used as an automated pick and place system in combination with a Pixy 2 Vision module.

Although it worked well, it had a very spartan human interface based on an LCD shield. The LCD shields are actually very nice for simple projects and prototyping, but when your project requires an extensive menu system the huge amount of button presses required to get to the setting you are looking for can get a bit annoying.

The solution to this problem was to fit a touchscreen.

In this article I will go though the steps to make the new controller, including the process of making a menu in the Nextion editor, creating the Arduino code, making a dedicated enclosure, assembly and some basic user instructions for the new controller.

A video of this project can be found on YouTube though the link below:

In my search for an Arduino compatible touchscreen, I came across the Nextion module. The Nextion touchscreens are available in several sizes, ranging from 2.4 to 10.1” and there are several product lines with different feature levels. For this project I opted for the Basic version with a screen size of 3.5”, which costs around 40 Euros. The Nextion display is connected to one of the serial ports on the Arduino for two-way communication.  I prefer this approach over a shield module since you are more flexible in designing an enclosure and it leaves all other Arduino pins accessible for other uses.

There is an Arduino library available, which makes it quite easy to read user input data from the display and also to write back values to the screen or change button states.

One might think that a graphical interface takes a lot of resources from the Arduino, but this is not the case. The graphical interface is fully driven by the on-board controller on the Nextion display. The only thing the Arduino has to take care of is the serial communication.

 Let’s start with making a menu system for the Nextion Display. Download the Nextion Editor and the Arduino Library from the Nextion website:

Editor: https://nextion.tech/nextion-editor/#_section1

Nextion editor

The Nextion editor is free software, which allows you to build the graphical interface for your project including buttons, images and built in code where needed. I will go through the design of the interface  a fairly high level in an attempt to keep this article short and readable, but if you need more information on how to use the Nextion Editor I would recommend the InterlinkKnight youtube channel, which contains several videos on how to use the editor and the touchscreen.
https://www.youtube.com/user/interlinkknight

After starting a new project where you indicate the type of display and the orientation, your project is opened in the main editor.

On the left side components like buttons and text fields can be added to the active screen.

The Picture window on the lower left can be used to import pictures for use in the background or to indicate the current state of a button. In this project I used this to place an open and close icon on the gripper button.

The output window shows messages from the editor, for example when compiling your code. This will allow you to debug your code.

In the event window you can add actions or your own code for any element you place on your screen, for example a button. The types of actions are dependent on the type of object you have selected.

The Page window on the right allows you to add different pages to your project to keep things organized.

In the Attribute window you can change properties of the selected component like size, color, text, etcetera.

In this project the regular buttons are used for actions like moving the robot arm or changing to a different menu page.

For example, let’s look at the “+” button on Jog menu Page. This button sends it’s ID to the Arduino via the serial port, so the Arduino knows the button was pressed and can initiate the desired action, in this case: move the robot arm. For this button, which has object name “b7”, I placed a checkmark in the “touch release event” tab. When the button is pressed and released, the nextion sends a touch release event to the Arduino for button “b7”.

Some other buttons on this page are dual state buttons, for which the name starts with “bt” instead of “b”. For example the “X” axis selection button. For this button I also wanted the Arduino to know it was pressed, so also in this case I placed a checkmark at the “send component ID”. However, I also want to deselect all other axes when this button is pressed, since you can only jog one axis at a time. This is achieved by the code under the “Touch Press Event” tab. It uses an IF statement, that depresses all other axes buttons when this button is pressed.

Note that the Nextion editor can be quite finicky about the syntax, for example if there is a space where it does not expect it, which can be quite confusing to the user. So if you get any errors from the compiler, look at some other code that DOES work and try to format it the same way.

For each page I also used some code for the preinitialize event to let the Arduino know which page is selected. This is done with the “printh” command. This allows the Arduino for example to update any text fields in the current page when it sees a new page is loaded. In this case it is used to populate the yellow coordinate text fields when the page is first loaded.
The “printh” command actually bypasses the Nextion arduino library. I have used this method in several locations in the program. You can choose whichever method works best for you.

For most buttons I simply used colours to indicate their state. Blue is depressed and green is pressed. This behaviour can be changed with the “bco” and “bco2” parameters.

Icon Design for Gripper Button

However for the vacuum button I wanted to use an Icon to indicate the state of the vacuum cup or gripper. To achieve this I made a bitmap in inkscape with the same aspect ratio as the screen, which is 3:2. The size is not important at this stage, as long as the ratio between width and height is 3:2. I placed this on top of a screenshot made from the from menu page in the Nextion editor. If you align the screenshot with the rectangle and change the opacity of the rectangle you can see the position of the button for which you want to design an icon.

You can then draw the icon, change the opacity back to 100% and export only the rectangle with the icon as a bitmap. This can be done by pressing Ctrl+shift+E. Change the image size to 480:320 and press “Export as”. Now do the same thing for the open state of the button, so you have 2 images: one with a blue open icon and one with a green closed icon.

Back in the Nextion editor import both images by pressing “add” in the “picture” window. Each picture will get an ID number assigned by the editor, in this case “0” and “1”. Next, you can select the correct picture for the button by entering the ID number in the “picc” and “picc2” fields.

Menu Pages

This project contains 6 pages to display the various menus. There are 4 main menus:

• Jog: Manually jog the robot arm to any position and operate the gripper or vacuum cup.
• Auto: Start the automatic pick and place cycle using input from the Pixy2 camera
• Settings: Here you can go to several predefined positions or change the end effector type. It also provides access to sub menus for setting speed, acceleration and delay times, as well as a sub menu with a QR-code which links back to my website.
• Calibration: This is used to calibrate the vision system, which is needed to translate coordinates between the camera and the robot arm. In the previous post I covered this more in detail. Also in this menu you can set the locations of the bins where the various parts should be dropped.

Changing between menu pages is done by assigning a “page” command to each button under “touch release event”. This lets the Nextion know it needs to show a specific page. The same method is used for entering sub menus and returning to the main menu through the “Back” button.

Just a couple of notes on the slider components used in the speed menu. For each slider I send a touch release event for both the “touch press event” as well as for the “touch move” event. This allows the Arduino to update the number field above the slider in real time when the user is touching or moving the slider.

In the editor the behaviour of the menu system can be evaluated before uploading it to the display by pressing “Debug”. This will bring up a simulator where you can press buttons, to see if they respond as intended. You can now actually switch between pages using the menu buttons.

Also the simulator generates the output that the actual display would send out through the serial port. This is very useful for debugging purposes. Of course any changes made by the Arduino to the state of display components is not captured here, but you can simulate this by entering commands the Arduino would send in the “instruction input area”.

When it is time to test the menu on the actual Nextion display the code can be uploaded via serial connection of micro SD card. Only the SD card option is covered in this article. Press Compile and verify if there are no error messages. Select “TFT file Output” under the file menu. This writes a TFT file to the micro SD card plugged into your PC. This single file is all that is required to update the Nextion with your project. Disconnect the Nextion from power and the serial connection. Place the the SD card in the nextion and turn the display back on by connection it to 5V power.

Just to be safe do not connect the serial wires yet. Wait for the update process to complete, unplug the display again, remove the SD card and now connect both the power connections and the wires for the serial connection.

I typically reset the Arduino after reconnecting the Nextion display just to make sure we have a clean start.

Arduino program

To enable communication between the Arduino and Nextion Display download arduino library.

Arduino Library: https://nextion.tech/nextion-editor/#_section3

In order to assign the nextion display to the desired serial port, open the Nexconfig.h file under the Arduino libraries folder.
This is by default under: C:\Program Files (x86)\Arduino\libraries\Nextion\NexConfig.h

Editing the file is needed to assign the Nextion display to serial 2 for this project (pins 16:TX2 and 17:RX2). Otherwise the library for the Nextion will not function properly.
Search for the following line and make sure the port is set to “Serial2”:
#define nexSerial Serial2

Arduino Mega serial ports for this project are as follows:

• Serial for Debugging messages on serial monitor of the Arduino IDE program. I guess you could call this serial0, but in the naming convention it is just called serial.
• Serial1 for communication with Dobot Magician, which used pins 18:TX1 and 19:RX1, for transmitting and receiving data respectively.
  Serial2 for communication with Nextion. This uses pins 16:TX2 and 17:RX2

Here is the wiring diagram for this project:

As a factory default, the Nextion display uses a baud rate of 9600. This is fine for most purposes but may give issues with slider elements.

I this case we will change to baud rate of 115200, which is also used for the other ports on this project.

This can be done from either inside the Arduino program or you can change the setting using the Nextion Editor.

Option1: Baud rate setting with arduino program:

// uncomment following section to set baud rate to Nextion display
/*
Serial2.begin(9600);  //pins 16 and 17 on Arduino Mega board, with Nextion display attached. //9600 is default baud rate for Nextion.
delay(500);  // make sure port is ready for next command
//set baud rate for Nextion to 115200 permanently.
//(use “baud” for temporary and “bauds” for permanent. In latter case you can remove command after executing once)
Serial2.print(“bauds=115200”); 
//always send this line 3 times after each commend to Nextion display
Serial2.write(0xff);  Serial2.write(0xff); Serial2.write(0xff);
delay(500); 
Serial2.begin(115200); 
*/
// end of baud rate section

Option2: baud rate setting using Nextion Editor:

Type comment “bauds=115200” under postinitialize event of the first menu page.

Note that I left the default baudrate at 9600 in the end because I sometimes had issues with the higher baud rate (The display sometimes reset itself to lower boud rate). In the end, I think for this project the higher baud rate was not strictly necessary.

Arduino Program basics

I think it is best to start with a very simple program as an example to show how the Arduino code should be created to work with the Nextion display.  In this example the onboard LED of the Arduino is turned on or off by pressing buttons on the Nextion display.

• At the start of your program the Nextion Library is included.
• Each screen object used in the Arduino program has to be declared. The statement includes the page number, object Id number and object name.
• Each object monitored by the Arduino shall also be included in a touch event list.
• Furthermore, each object has its own function, which is called when a button is pressed. Here you place the code that you would like to run if a specific button is pressed on the display.
• The various components are then linked to the functions by means of the attachpop command. This lets the Arduino know which function should be called for each button.
• Finally the void loop function contains the nexloop command, which will read the serial port each cycle and relay the commands to the Arduino.

#include <Nextion.h>
const int led = 13;

NexButton b0 = NexButton(0,3,”b0″);
NexButton b1 = NexButton(0,4,”b1″);

NexTouch *nex_listen_list[] = {
  &b0,
  &b1,
  NULL
};

void b0PopCallback(void *ptr){  digitalWrite(led,HIGH); }
void b1PopCallback(void *ptr){  digitalWrite(led,LOW); }

void setup(void) {
 b0.attachPop(b0PopCallback,&b0);
 b1.attachPop(b1PopCallback,&b1);
  pinMode(led,OUTPUT);
  digitalWrite(led,LOW);
}
void loop() {
  nexLoop(nex_listen_list);
}

The Arduino controller program

The program for the Dobot controller contains the same type of commands as mentioned in the example. Since there are quite a few buttons and other components on the display there is a lot more code involved, but for each item the approach is exactly the same.

If you would like to see or use the code it can be downloaded from github, which is also linked below. In this repository also the files for opening this project in the Nextion editor are included.

https://github.com/Robin61/Arduino-Nextion-Controller-for-Dobot-Magician-Robot-Arm

Conserving Arduino memory

Because I was writing a a large number of messages to the Arduino serial monitor for debugging the compiler showed an error message: “low memory available, stability issues may occur”. I did started noticing erratic behaviour from the Arduino, to a point where it became completely unresponsive.

There is actually a surprisingly easy fix for this:
You can use the “F” function in Arduino code to reduce memory usage and store strings in flash memory instead of the limited SRAM:
for example:
Change Serial.print(“This is a test message”)
to: Serial.print(F(“This is a test message”))

The ATmega2560 has following memory space :
Flash – 256kB (of which 8kB is used for the bootloader)
SRAM – 8kB
EEPROM 4kB
So, there is 32 times more Flash memory available than SRAM (not counting the 8kB used for the bootloader). Using the “F” function makes a significant difference in memory usage and allows you to keep the debugging text messages in your project, which to me are very useful.

Case for the Arduino and Nextion display

I designed a case for this project, which holds both the Arduino Mega and the Nextion display. The case has a form-follows-function design. The intention here was to make a practical case that that can be used for multiple projects. There is enough room to place the dupont connectors in the Arduino headers underneath the lid.

The CAD model for the case can be downloaded from GrabCAD using the link below:
https://grabcad.com/library/case-for-arduino-mega-and-nextion-3-5-touchscreen-1

On the bottom several standoffs are available, for mounting the Arduino Mega. The standoffs are meant for M3 screws. The holes do not have a thread. Instead, the holes are undersized, allowing the M3 screws to basically act as self tapping screws. This creates a very strong fixation, but it does not lend itself to frequent tightening and untightening. This would quickly wear out the threads.

The same principle is used for the mounting holes of the Nextion display in the lid. The only place where I used threaded inserts is for mounting the lid onto the case, since you might frequently take of the lid when working on a project. The threaded inserts are the cheap kind you can find on AliExpress and not really a good design for using in 3D printed parts. However, these are the only ones I have available. The inserts are pressed into the lid by using a soldering iron.

When using a standard soldering tip, you have the chance of pulling the insert back out, since it is easy to get it stuck on the tip. A flat piece of metal can be used to prevent this and leave a nice flat surface at the same time. It is difficult to get the inserts in perfectly straight, but can be corrected later if necessary. Make sure not to over tighten the screws with this type of insert, since there is a risk of pulling it out.

To connect the Dobot to the Arduino I made a flat-cable with Dupont connectors on one end. There are other options of course, but this gave me the opportunity to practice crimping Dupont connectors, which turned out to be easier than expected. The connectors should be crimped with the smallest cavity, but you can start off with a larger cavity in which the open connector fits.

Then pull the wings of the connector until they bottom out in the cavity. Close the crimp tool until the connector is fixed, but still able to accept the wire. Place the wire into the connector and fully close the crimp tool.

After this step you can place the connector in the smallest cavity and close the tool again. The connector can now be inserted into the plastic sleeve. Job done.

I do typically check the wire with a multimeter to see if the connection was made properly.

There are a couple of wire clamps which can be used for cable management of jumper wires and flat-cables. On the left side of the case there is a wire port which is open at the top, for routing wires to the outside of the case even when they are already connected. A small strip can be used to keep the wires in place.

When the components and wires are in place, the lid is placed on top of the case and tightened from the back with M3 screws.

Controller operation

So, let’s go through the menu of the new controller. The Jog menu has buttons to control all axes of the robot arm and a button to operate the end effector, in this case a vacuum cup.

After selecting an axis you can select the jog increment and move the axis in the positive or negative direction. The current positions are displayed in the yellow text fields and updated each time the arm moves.

The Auto menu has just 2 buttons, one to start the automatic cycle and one to stop the cycle. Also there are 2 fields displaying the X and Y coordinate of the part selected by the Arduino to be picked up from the table.

In the Automatic cycle the program goes through a continuous loop, looking for part sugnatures. If a part is found, it will be picked up and dropped in the correct pre-defined location. When the table is empty the automatic cycle will be terminated.

The settings menu has buttons to save or move to the camera position, start position and z-down position. The buttons are color coded. The blue ones execute a command and the orange buttons are saving a setting to the EEPROM.

The radio buttons can be used to select the type of end effector.

Under the submenu called “speed” there are several sliders. The first slider controls the motion speed of the arm. The second slider controls the acceleration, so how long it takes for the arm to reach the programmed speed. The delay time slider determines how long the program waits before sending another command to the robot arm.

Using a delay timer is not the best option, but I have not been able to get the position feedback working yet. A very helpful member of the Dobot forum provided me with modified libraries for this purpose, but I was not able to fully understand how they worked, so I decided to stick with sub optimal code that I was able to integrate into my project.

The settings in this menu can either be accepted for single use by pressing the “back” button or saved to EEPROM by pressing the orange button. There is also an about menu, which contains a QR code leading to my website, which would basically serve an online user manual for the controller. Adding a QR code is very easy in the Nextion software. Just place it on the screen and type in a hyperlink.

Next up is the calibration screen. This contains buttons for moving to the various positions or saving them in the EEPROM.

The large start button starts the calibration routine where the camera registers the location of the calibration dots. After this is done the user should manually go to each of the calibration dots with the end effector and store its position to complete the calibration procedure.

For each calibration dot you can use the jog menu to go physically to its location, after which the location can be stored in the EEPROM memory by pressing the save button in the calibration menu. Please refer to my previous article for more detailed background information on the calibration procedure.

Below are a picture of the robot arm in the pick and place cycle, as well as a typical screenshot from the Pixy 2 camera at the capturing location and the serial monitor, which is used for debugging:

Final thoughts

A touch screen can be quite useful for an offline robot controller. It makes controlling the robot easier and provides more screen real estate to display information or include additional buttons. Creating a proper graphical interface takes some time, but once you have some experience with the Nextion Editor software, it is (for me) a lot faster than programming a character based LCD menu. I think the touch screen displays can be useful for a variety of projects. For smaller projects the price can be a deal breaker since at 40 Euros it costs around 4 times as much as an LCD shield.

A possible (ergonomic) improvement for the Dobot controller is to create a box that is more suitable for hand held operation.
If you have any questions or comments please leave a comment below and I will do my best to help.

Cheers,
Robin

References

Arduino Code and Nextion files for this project can be found on my Github page:

https://github.com/Robin61/Arduino-Nextion-Controller-for-Dobot-Magician-Robot-Arm

The CAD model for the case can be downloaded from GrabCAD:
https://grabcad.com/library/case-for-arduino-mega-and-nextion-3-5-touchscreen-1

Instruction set for Nextion Display: https://www.itead.cc/wiki/Nextion_Instruction_Set

Useful videos on the Nextion display by Interlinkknight:
Troubleshooting Nextion projects and setting serial baud rate: https://www.youtube.com/watch?v=oJuL0hNk2eU
InterlinkKnight Youtube channel –  Example Nextion menu with multiple pages:
https://www.youtube.com/watch?v=mdkUBB60HoI

Example arduino programs for Nextion: https://github.com/itead/ITEADLIB_Arduino_Nextion