MQTT to Google IoT

#1

Hi,

I have a working system based on the excellent MQTT article and spreadsheet.

I can send the data via hex packets to publish and subscribe to topics on CloudMQTT and Ubidots. I can test using MQTTlens. It all works perfectly.
I am using PIC24FJ64 and a SIM5300E (though plan to change to SIM868 for cost reasons)

Now my customer gives me some extra problems. They want a more professional implementation of MQTT (CloudMQTT works for me but their servers have been unreliable and the documentation is poor).
Also the customer is not familiar with the connection at his side (servers) and MQTT brokers. In fact they do not like the idea of a 3rd party broker.

They have asked if it is possible to use Google IoT
https://cloud.google.com/iot/docs/how-tos/mqtt-bridge
But already I see things that I do not recognise like JWT web tokens for the password:
https://cloud.google.com/iot/docs/how-tos/credentials/jwts

Is this possible to achieve? Perhaps it is too much to ask from the low cost module and the microcontroller in C?
Thank you in advance for any help.

#2

Hi Patrick,
Glad that you got it working.
Am trying to use Google IOT but the problem is it only supports encrypted connections.
You need to encrypt data using RS256 encryption. JWT, you can generate by using JWT.io or you can look at the example code in Google docs for Node.js and Python and C. Google has good example for HTTP clients in above languages. I havent tried C though.
But first step would be to get the encryption working on your PIC MCU as Google doesnt support non secure connections.
You can also use open source broker EMQ installed on your own cloud instance and use it. I have made a video recently on this. Check it out.

#3

Thanks,

It sounds quite challenging to use the encryption on the PIC but I will try.
I found this, perhaps it is interesting for you as well.
http://www.microchip.com/design-centers/security-ics/cryptoauthentication/cloud-authentication/google-iot-core-atecc608a

The problem I have is that the web developers dont want to make any effort. They think PICs are like Raspberry Pi!
I will let you know how I get on.

#4

The customer has now specified that the connection should be made via MQTT to Azure IoT.
This is not a standard MQTT broker so I am having some challenges to make the connection.

I have created my Azure IoT hub and device. I am able to test the conncection using the “Device Manager Tool” and 3rd party simulation software MQTTbox. I am able to create the SAS token. This connection works well passing information both ways.
So I know the connection is correct and my SAS is valid.

I am following the Azure documentation “Using the MQTT Protocol Directly”

The first important thing I notice is that Azure will only allow secure connections:

To use the MQTT protocol directly, your client must connect over TLS/SSL. Attempts to skip this step fail with connection errors.

In order to establish a TLS connection, you may need to download and reference the DigiCert Baltimore Root Certificate. This certificate is the one that Azure uses to secure the connection. You can find this certificate in the Azure-iot-sdk-c repository. More information about these certificates can be found on Digicert’s website.

I am unsure how to acheive the secure connection on the SIM5300E but I have included the command
“AT+CIPSSL=1” and I get the “OK” response.

I connect using the command:
AT+CIPSTART=TCP",“xxxxxxxx.azure-devices.net”,“8883” (where xxxxxxx is my IoT hub name)
I get “CONNECT OK” response

I then get the “>” prompt when I send “AT+CIPSEND”.

I have carefully constructed the Connect packet according to the documentation. I send it using the same code as I have used successfully with CloudMQTT but the server closes the connection with no response.

My questions at this stage are:
(1) Does the “CONNECT OK” response mean that I am connected over SSL? Or is it possible that this is still the problem?
(2) It could be that my Connect Packet is not correct, even though I have checked it. Has anyone been able to connect directly to the Azure over MQTT.
(3) What could be the cause that I do not receive any response except “CONNECT OK” then "CLOSE OK?

#5

Update Information:
I found in the AT command guide a useful command “AT+CIPACK” I get the response after I send my Connect Packet:

AT+CIPACK +CIPACK: 230,230,0

OK

My calculated “Remaining Length” is 228. Add the “0x10” start byte and the “Remaining Length” byte itself and that makes 230. So that sounds correct.

From the SIMCOM manual:

The command AT+CIPACK is used to query previous connection data transmitting state. In single connection, the execution command AT+CIPACK will return +CIPACK: ,, . The first parameter is the data amount which has been sent; the second parameter is the data amount confirmed successfully by the server; and the third parameter is the data amount without confirmed by the server. As long as the connection is still active, user can know how much TCP data user sent to server and how much is received successfully by the server in total. By this means, user can get the total data transmitting amount.

Perhaps this is an indication that the SSL connection is working?
Though I still have no response from the Azure server.

#6

Update - I have success. SIM5300E has SSL over TCP but not HTTPS. TCP is fine for MQTT to Azure.

#7

Thanks for updating Patrick. We would be waiting to hear from you soon :slight_smile:

#8

Update… SIM868 also seems to have TCP over SSL as standard, though I have not fully tested it to Azure MQTT.

#9

Hi Patrick,
Can you please share the steps followed to send data over encrypted TCP/MQTT connection. Did you generate the keys needed for it?

#10

Hello,

I am also trying to create connect packet for microsoft azure and after pushing packets I am getting 32,2,0,1 as response in bytes
What it means ? whether it is connected or not ?

#11

Hi Parth,

Id this in hex or decimal?

32 would make no sense as a packet header for a Connack:
http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#Figure_2.2-

#12

Its in bytes[4] = {32,2,0,1}
After Connect Packet I am passing publish packet and as I can see message is not getting published
Can you cross check my publish packet
0x30
, 0x28 //RL
, 0x00
, 0x24 //Topic Length [ Topic for Azure : devices/EnergyMeter/messages/events/]
, 0x64
, 0x65
, 0x76
, 0x69
, 0x63
, 0x65
, 0x73
, 0x2f
, 0x45
, 0x6e
, 0x65
, 0x72
, 0x67
, 0x79
, 0x4d
, 0x65
, 0x74
, 0x65
, 0x72
, 0x2f
, 0x6d
, 0x65
, 0x73
, 0x73
, 0x61
, 0x67
, 0x65
, 0x73
, 0x2f
, 0x65
, 0x76
, 0x65
, 0x6e
, 0x74
, 0x73
, 0x2f
, 0x48 //H
, 0x69 //i

#13

Here is some of my connect code to assist. It is based heavily on Ravi’s previous MQTT examples so thanks to him for this.
Where you see XXXXXX your own details go. The password you need to generate using tokens yourself for each device.

One other thing I have learned is that the “Keep Alive” seems to have a maximum of 2 minutes but I am investigating this.

const char MQTTClientID[20] = “XXXXXXXX”;
const char MQTTSubscribeTopic[100] = “devicesXXXXXXXXX/messages/devicebound/#”;
const char MQTTPublishTopic[100] = “devices/XXXXXXX/messages/events/”;
const char MQTTUsername[100] = “XXXXXXX.azure-devices.net/XXXXXXXX”;
const char MQTTPassword[200] = “SharedAccessSignature sr=XXXXXXX.azure-devices.net%2Fdevices%2FXXXXXXXXX&sig=95B790Gi5KhxdwY2QHxQroVslUsrEz7P4aVFkezDe%2FQ%3D&se=1534752364”;

void connect_MQTT(void)
{
SIMCOM_MODE = AT_COMMS;
//Calculate MQTT Lengths
MQTTProtocolNameLength = strlen(MQTTProtocolName);
MQTTClientIDLength = strlen(MQTTClientID);
MQTTUsernameLength = strlen(MQTTUsername);
MQTTPasswordLength = strlen(MQTTPassword);
datalength = MQTTProtocolNameLength + 2 + 4 + MQTTClientIDLength + 2 + MQTTUsernameLength + 2 + MQTTPasswordLength + 2;

//Get Prompt
__delay_ms(10);
__delay_ms(1);
putsU2("AT+CIPSEND"); //Send the command to 5300E
__delay_ms(200);
__delay_ms(1);

//Send Byte - Declare as connect
putU2(0x10); //0x10 = Fixed value determined by MQTT Standard
//Send Byte - remaining Length  -- Calculated Manually 

X = datalength;
while (X > 0)
    {
    encodedByte = (X % 128);
    X = (X / 128);
    // if there are more data to encode, set the top bit of this byte (to indicate more data)
    if (X > 0)
        encodedByte = (encodedByte | 128);
    putU2(encodedByte);
    }

//Send 2 Bytes - Protocol Length
putU2(MQTTProtocolNameLength >> 8);
putU2(MQTTProtocolNameLength & 0xFF);
//Send Bytes - Protocol Name
for (Counter = 0; Counter < MQTTProtocolNameLength; Counter++)
    {
    putU2(MQTTProtocolName[Counter]);
    }

//Send Byte - Level
putU2(MQTTLVL);
//Send Byte - Flags
putU2(MQTTFlags);
//Send 2 Bytes - Keep Alive
putU2(MQTTKeepAlive >> 8);
putU2(MQTTKeepAlive & 0xFF);

//Send 2 Bytes - Client ID Length
putU2(MQTTClientIDLength >> 8);
putU2(MQTTClientIDLength & 0xFF);
//Send Bytes - (MQTTClientID);
for (Counter = 0; Counter < MQTTClientIDLength; Counter++)
    {
    putU2(MQTTClientID[Counter]);
    }
//Send 2 Bytes - MQTTUsernameLength
putU2(MQTTUsernameLength >> 8);
putU2(MQTTUsernameLength & 0xFF);
// putU2(0x00);
// putU2(0x00);


// Send Bytes -(MQTTUsername);
for (Counter = 0; Counter < MQTTUsernameLength; Counter++)
    {
    putU2(MQTTUsername[Counter]);
    }
//Send 2 Bytes - MQTTPasswordLength
putU2(MQTTPasswordLength >> 8);
putU2(MQTTPasswordLength & 0xFF);
// Send Bytes - MQTTPassword
for (Counter = 0; Counter < MQTTPasswordLength; Counter++)
    {

    putU2(MQTTPassword[Counter]);
    }

//Send Byte - CTRL-Z
putU2(0x1A)
}
#14

Followed and As I can see connack packet 2,0 in my bytes seems it is connected.
I think problem is in publish packet.
I am using C#.Net.

Keep Alive is 0x3c in my case. 2 minutes means 0x78 right ?

#15

Pub ack should have a header 0x40

The code I use is nearly identical to Ravi’s

void Publish_MQTT(void)
{
SIMCOM_MODE = AT_COMMS;
//Calculate MQTT Lengths

MQTTPublishTopicLength = strlen(MQTTPublishTopic);
MQTTPublishPayloadLength = strlen(MQTTPublishPayload);
datalength = (2 + MQTTPublishTopicLength + MQTTPublishPayloadLength + 2);

//Get Prompt
__delay_ms(10);
__delay_ms(1);
putsU2("AT+CIPSEND"); //Send the command to 5300E
__delay_ms(200);
__delay_ms(1);

//Send Byte - Declare as publish
putU2(0x32); //Fixed value determined by MQTT Standard
//Send Bytes - remaining Length  

//putU2(datalength);
X = datalength;
while (X > 0)
    {
    encodedByte = (X % 128);
    X = (X / 128);
    // if there are more data to encode, set the top bit of this byte (to indicate more data)
    if (X > 0)
        encodedByte = (encodedByte | 128);
    putU2(encodedByte);
    }

//Send 2 Bytes - Topic Length
putU2(MQTTPublishTopicLength >> 8);
putU2(MQTTPublishTopicLength & 0xFF);
//Send Publish Topic
for (Counter = 0; Counter < MQTTPublishTopicLength; Counter++)
    {
    putU2(MQTTPublishTopic[Counter]);
    }

//Send Bytes - Packet ID
putU2(0x00);
putU2(0x03);

//Send Publish Payload
for (Counter = 0; Counter < MQTTPublishPayloadLength; Counter++)
    {

    putU2(MQTTPublishPayload[Counter]);
    }
//Send Byte - CTRL-Z
putU2(0x1A);

ListenServer(5);
SIMCOM_MODE = MQTT_STANDBY; //Revert the comms mode
}
#16

On publish server returns nothing.
Please check my publish packet. Can you tell me whether it is wrong ?
Please Help i am stuck…you said add header 0x40 but as your code suggest I have to add 0x32

#17

Try with QoS set to 1.
Azure does not like it set to 0.

#18

Changing 0xC2 to 0xCA for Qos1 results same for connect packet 32,2,0,1
Have converted bytes to Hex for 32,2,0,1 it is 0x20-0x02-0x00-0x01

Can you please take online session and have a look whats happening in my case ?

#19

0xC2?
Header should be 0x32 like in my example.

0x20-0x02-0x00-0x01 looks like a valid Connack response from server.
Whats the Puback response?
it may just close the connection if its malformed.

#20

But In my connect response as per below link
0x01 is not good

Puback response is nothing empty