-

Fetch and parse JSON on ESP8266 wifi module :

Use case:
 I want monitor particular json stored on server which can be modify by website from remote distance.
It can be useful where you can modify control bits from website and change GPIO status at ESP8266 side.

1) Home automation
2) Information parsing
3) Staus monitoring
4) non realtime notification

This system work on principle on monitoring files.

Feature :
  Stand alone operation of ESP8266.
  Uses JSON parsing so data extraction is very easy.

Required :
1)ESP8266
2)Arduino or any USB to serial converter with 3.3v or 5v power connection pin out.

Basic Working:
  
  • ESP8266 : 
It uses the local router to connect internet and fetch JSON from server using HTTP get request after every second parse it and display JSON variables on serial monitor(you can compare them and change GPIO for device control).

Shown in diagram it send GET request at server and server send back HEADER + JSON to ESP8266
HTTP request :
 client.print(String("GET ") + url + " HTTP/1.1\r\n" +
                 "Host:" + host + "\r\n" +
                "User-Agent:Mozilla/5.0(Windows NT 6.3; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0" +
                "Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" +
               "Connection:close\r\n\r\n");

 Server Responce:
HTTP/1.1 200 OK
Vary: Host,Accept-Encoding
P3P: CP="Not supported"
Set-Cookie: _xta_uid=4945816047688c1c34e113b0f9c163b0; expires=Thu, 27-Sep-2018 06:01:09 GMT; Max-Age=63072000; path=/; domain=.mobie.in; httponly
Content-Type: text/plain;charset=UTF-8
Content-Length: 530
Accept-Ranges: bytes
Date: Tue, 27 Sep 2016 06:01:17 GMT
X-Varnish: 644140890
Age: 0
Via: 1.1 varnish
X-Robots-Tag: Crawl-delay: 10

{
   "id": 1,
   "name": "Leanne Graham",
   "username": "Bret",
   "email": "Sincere@april.biz",
   "address": {
     "street": "Kulas Light",
     "suite": "Apt. 556",
     "city": "Gwenborough",
     "zipcode": "92998-3874",
     "geo": {
       "lat": "-37.3159",
       "lng": "81.1496"
     }
   },
   "phone": "1-770-736-8031 x56442",
   "website": "hildegard.org",
   "company": {
     "name": "Romaguera-Crona",
     "catchPhrase": "Multi-layered client-server neural-net",
     "bs": "harness real-time e-markets"
   }
}

to separate only data and neglect header i used
while(client.available()){
    line = client.readStringUntil('\r');
    Serial.print(line);
  }

this code store data in 'line'  untile received data ends with '\r' ,
we received data as  [ header + \r\n\r\n + data(json) + \r\n\r\n ]
so first instance "line" contain header, As next data received "line" override with data(json).

 this data is given to the JSON pacer along with user define structure to store in local variables.
  • Server :
          Stores the html page, json file and jsp scripts. Server gives access to the ESP8266 and remote client. Server can execute the jsp script as per remote client request to modify JSON file. this is not implemented yet.
  •  Remote Client : 
         Can access webpage that contain control panel or GUI for modify JSON file as per requirement here in this tutorial I have not implemented this.

Step 1:
Follow this post to setup ESP8266 programming environment with Arduino IDE.

Step 2 :
Register at any free web hosting site  or you can have your own domain or you can test it with your local server.
I used http://xtgem.com to host test.json file.
Register -
Create New Site - here i created "codersmagic.mobie.in"
 

Step 3:
Create JSON file
for create json file you can use any online json creator. Past json data in simple txt file and rename it as JSON.

Note : before save json file confirm that it don't contain any hidden '\r' in it.  '\n' is ok.

Here I used Notepad++ to replace '\r' with non-value means erase all '\r'
 
 save test.json file.

Step 4 :
Goto xtgem.com > login > click on your site > upload test.json file to '/' root directory
 
 here I alredy uploaded JSON file.
confirm that json file is accessible in web browser from URL given in right side corner link.

Step 5: 
 Downlod Arduino json parsing librabry from given url https://github.com/bblanchon/ArduinoJson . install it in Arduino IDE from Sketch > include library > Add .ZIP library...

Now upload following sketch in ESP8266.

#include
#include

const char* ssid     = "MyRouter";
const char* password = "MyRouterPassword";

const char* host = "codersmagic.mobie.in";
const int httpPort = 80;
char charVar[15];

struct UserData {
  int id;
  char name[32];
  char username[32];
  char email[32];
  struct{
          char street[32];
          char suite[32];
          char city[32];
          char zipcode[32];
           struct{
                 char lat[10];
                 char lng[10];
            }geo;
  }address;
  char website[32];
  char company[32];
};

UserData userData;

void setup() {
  Serial.begin(115200);
  delay(10);

  // We start by connecting to a WiFi network
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);

  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(300);
    Serial.print(".");
  }
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
}

void loop() {
  delay(2000);
  String line;
 
  Serial.print("connecting to ");
  Serial.println(host);

  // Use WiFiClient class to create TCP connections
  WiFiClient client;

  if (!client.connect(host, httpPort)) {
    Serial.println("connection failed");
    return;
  }

  // We now create a URI for the request
  String url = "/test.json";

  Serial.print("Requesting URL: ");
  Serial.println(url);

  // This will send the request to the server.
  client.print(String("GET ") + url + " HTTP/1.1\r\n" +
                 "Host:" + host + "\r\n" +
                "User-Agent:Mozilla/5.0(Windows NT 6.3; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0" +
                "Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" +
               "Connection:close\r\n\r\n");
 
  delay(10);
 
  // Read all the lines of the reply from server and print them to Serial
  while(client.available()){
    line = client.readStringUntil('\r');
   // Serial.print(line);
  }
  Serial.println();
  Serial.println("----------------------JSON PARSER-------------------------");

  char* charBuf = (char*)malloc(sizeof(char)*1024);  //dynamic memory allocation avoid most of ESP watch dog exeptions
  line.toCharArray(charBuf,1024);
      if (parseUserData(charBuf, &userData)) {
        printUserData(&userData);
      }
  Serial.println("-------------------------------------------------------");
 
  free(charBuf); // remember to free dynamic memory
    

  Serial.println("closing connection");
}


bool parseUserData(char* content, struct UserData* userData) {
  // Compute optimal size of the JSON buffer according to what we need to parse.
  // This is only required if you use StaticJsonBuffer.
  const size_t BUFFER_SIZE =
      JSON_OBJECT_SIZE(8)     // the root object has 8 elements
      + JSON_OBJECT_SIZE(5)   // the "address" object has 5 elements
      + JSON_OBJECT_SIZE(2)   // the "geo" object has 2 elements
      + JSON_OBJECT_SIZE(3);  // the "company" object has 3 elements

  // Allocate a temporary memory pool on the stack
  StaticJsonBuffer jsonBuffer;
  // If the memory pool is too big for the stack, use this instead:
  // DynamicJsonBuffer jsonBuffer;

  JsonObject& root = jsonBuffer.parseObject(content);

  if (!root.success()) {
    Serial.println("JSON parsing failed!");
    return false;
  }
 
  //root.prettyPrintTo(Serial);   //print raw JSON data
 
  // Here were copy the strings we're interested in
  userData->id = root["id"];
  strcpy(userData->name, root["name"]);
  strcpy(userData->username, root["username"]);
  strcpy(userData->email, root["email"]);
  strcpy(userData->address.street, root["address"]["street"]);
  strcpy(userData->address.suite,root["address"]["suite"]);
  strcpy(userData->address.city,root["address"]["city"]);
  strcpy(userData->address.zipcode,root["address"]["zipcode"]);
  strcpy(userData->website, root["website"]);
  strcpy(userData->company, root["company"]["name"]);
 
  // It's not mandatory to make a copy, you could just use the pointers
  // Since, they are pointing inside the "content" buffer, so you need to make
  // sure it's still in memory when you read the string

  return true;
}


void printUserData(const struct UserData* userData) {
  Serial.println("");
  Serial.print("id = ");
  Serial.println(userData->id);
 
  Serial.print("Name = ");
  Serial.println(userData->name);

  Serial.print("User Name = ");
  Serial.println(userData->username);

  Serial.print("Email = ");
  Serial.println(userData->email);

  Serial.print("Adress = ");
  Serial.println(userData->address.street);

  Serial.print("Suite = ");
  Serial.println(userData->address.suite);
 
  Serial.print("City = ");
  Serial.println(userData->address.city);

  Serial.print("Zip Code = ");
  Serial.println(userData->address.zipcode);
  
  Serial.print("website = ");
  Serial.println(userData->website);
 
  Serial.print("Company = ");
  Serial.println(userData->company);
/*
  "id": 1,
   "name": "Leanne Graham",
   "username": "Bret",
   "email": "Sincere@april.biz",
   "address": {
     "street": "Kulas Light",
     "suite": "Apt. 556",
     "city": "Gwenborough",
     "zipcode": "92998-3874",
     "geo": {
       "lat": "-37.3159",
       "lng": "81.1496"
     }
   },
   "phone": "1-770-736-8031 x56442",
   "website": "hildegard.org",
   "company": {
     "name": "Romaguera-Crona",
     "catchPhrase": "Multi-layered client-server neural-net",
     "bs": "harness real-time e-markets"
*/
}

 Step 6:
 See the serial log for JSON data on baud rate 115200.

Connecting to MyRouter
........WiFi connected
IP address:
192.168.0.5
connecting to codersmagic.mobie.in
Requesting URL: /test.json

----------------------JSON PARSER-------------------------

id = 1
Name = Leanne Graham
User Name = Bret
Email = Sincere@april.biz
Adress = Kulas Light
Suite = Apt. 556
City = Gwenborough
Zip Code = 92998-3874
website = hildegard.org
Company = Romaguera-Crona
-------------------------------------------------------
closing connection



some time you get error like
----------------------JSON PARSER-------------------------
JSON parsing failed!
-------------------------------------------------------

It is due to frequently hinting the same URL, server ignore some request you can adjust delay between two request to avoid this issue. still it is not guaranty that the HTTP response will receive for every request.

If have any question please comment below.

Comments

  1. Unknown10 November 2017 at 18:32

    The header files at the top are left blank,would you please mention it

    ReplyDelete

Post a Comment

Popular posts from this blog

-
Image
JHD12864E interfacing with Arduino Uno( 128x64 Bit MAp Graphics LCD) :   Requirement : 1 ) JHD12864E LCD 2 ) Arduino 3 ) Connecting wires Aim: Convert and display JPEG image on Graphics LCD as Bitmap image Step 1 :  Download and put Glcd library in Documents/Arduino/library folder GLCD library Step 2 : Download BMP-To-LCD software to convert JPEG image to BMP array BMP-TO-LCD Step 3:   Make connection of LCD to Arduino UNO as shown in figure.   Step 4:  Download any image from internet.   Step 5: Open image in Paint   Step 6:   Step7 :       Step 8:   Step 9:  Click on resize without selecting area Step 10:  Save image as 1.bmp   Step 11 : open Image in BMP-LCD software And click on Genarate code Copy that genarated code in Arduino sketch   Step 12: Copy Paste code in bitmaps.h with some mod...
Read more
-
Image
Program ESP8266 WiFi module with Arduino IDE: The Arduino IDE software (short for Integrated Development Environment) is the foundation of what has made the Arduino platform so successful. With it, Makers can program a wide range of compatible microcontroller boards using Arduino’s relatively approachable programming language. Behind the scenes, the IDE software translates your instructions into a more complex code required for your board’s specific chip. Aside from generally cleaning up a number of known bugs, the new update introduces a number of new tools and features. One of the useful feature is board manager which allow use to add customized and diffrent board support in Arduino IDE. In this tutorial we are going to add ESP8266 board in Arduino IDE and program it without and programmer simple using RX, TX pin. Required : 1) ESP8266 2) Arduino UNO 3)connecting wires 4) Local wifi router Step 1 : For buy ESP8266 module here amezon link is given Buy ES...
Read more