SaiLog YDVR NMEA-2000 Traffic Analysis

Hi everyone,
Another technical post today.  We're continuing to keep ourselves busy under lockdown at Gosport Premier Marina.  In the preceding post we described how we plot passage data from our YDVR Voyage Recorder.  In this post we describe a Python script to analyse recorded NMEA-2000 traffic.  The objective is to better understand the traffic on our network.

Total Traffic Pie Charts

To view each pie charts plot in more detail click on it!

Our new python script SaiLog_Stat presents statistics calculated from the YDVR log file.  For our 40 hour recording the log file is 830MB in size.  The following paragraphs walk through the results produced by the script.

Basics

Initially SaiLog_Stat lists some basic characteristics of the traffic...

############################################################
#                       SaiLog_Stat                        #
############################################################
#         !!! All Statistics Are Approximate !!!           #
############################################################
#
# Loaded file ../n2k/ZenAgainN2K_London-Gosport.log
#
# Processing 9695591 NMEA2000 messages
#
############################################################
# Basics
############################################################
#
#   Start Time        = 2020-03-18 06:45:11
#   Stop Time         = 2020-03-19 21:44:57
#   Duration          = 140386 seconds
#   Message Count     = 9695591
#   Message Frequency = 69.1Hz (average)
#   Bit Rate          = ~8333bps (approx)
#   Bus Utilisation   = ~3.3 percent (approx)

The bus utilisation is reassuring!

Devices

The script then interprets Product Information messages in which each device describes itself on the network.  On our network we have:

• SPx5 = Raymarine Autopilot
• GPSmap451 = Garmin GPS ChartPlotter
• VHF300iAIS = Garmin VHF Transceiver (with its AIS receiver disabled)
• XB8000 = Vesper AIS transceiver (with GPS)
• WiFi Replicator = NMEA-2000 to WiFi Bridge
• DST200 = Airmar DST800 Speed/Depth Sensor
• WX Series = Airmar 220WX Weather Station

############################################################

# Devices
############################################################
#
# Device Address 0
#   Model             = SPx5  E12197
#   Software Version  = Raymarine SPxx v
#   Model Version     = E12197ne SPxx v
#   Model Serial Code = 0000000e SPxx v
#
# Device Address 1
#   Model             = GPSmap 451 AUS
#   Software Version  = 4.60
#   Model Version     = 1.0
#   Model Serial Code = 3820961182
#
# Device Address 2
#   Model             = VHF 300i AIS
#   Software Version  = 2.10
#   Model Version     = 2.00
#   Model Serial Code = 1MN000235
#
# Device Address 21
#   Model             = XB8000F
#   Software Version  = 5.20.17443
#   Model Version     = F,ECN2,HW3,BB467
#   Model Serial Code = KW75348
#
# Device Address 32
#   Model             = WiFi REPLICATOR
#   Software Version  = 1.5 (2018-09-09)
#   Model Version     = 1.1 (2018-09-09)
#   Model Serial Code = 238243780
#
# Device Address 35
#   Model             = DST200
#   Software Version  = 3.004,3.050
#   Model Version     = 44-196-1-01,HW2:
#   Model Serial Code = 3822649
#
# Device Address 36
#   Model             = WX Series Weathe
#   Software Version  = 2.006,2.300,2.00
#   Model Version     = 44-854-1-02,HW11
#   Model Serial Code = 60111222

It's interesting how poorly some manufacturers encode the Model field.  Some include non-printable characters which SaiLog_Stat removes.

Messages

Next the script counts occurrences of each message type and accumulates their sizes, for each device.  Count, percentage, frequency, bitrate and name are tabulated for each message type...

############################################################
# Messages
############################################################
#
# Device "SPx5  E12197" (2254435 messages, 18038000 bytes, ~1124.3bps)
#      Count     %    Freq   Bitrate   Name
#         33   0.0   0.0Hz    0.0bps   ISO Address Claim                       
#         20   0.0   0.0Hz    0.2bps   Product Information                     
#     749613  33.3   5.3Hz  373.8bps   Rudder                                  
#         51   0.0   0.0Hz    0.0bps   Vessel Heading                          
#
# Device "GPSmap 451 AUS" (2310529 messages, 46443426 bytes, ~2745.4bps)
#      Count     %    Freq   Bitrate   Name
#      20908   0.9   0.1Hz    4.5bps   ISO Request                             
#         29   0.0   0.0Hz    0.0bps   ISO Address Claim                       
#          4   0.0   0.0Hz    0.0bps   ISO Manufacturer Proprietary A          
#     409044  17.7   2.9Hz  437.1bps   NMEA Group Function                     
#     245373  10.6   1.7Hz  518.5bps   Manufacturer Proprietary C              
#         17   0.0   0.0Hz    0.1bps   Product Information                     
#      81758   3.5   0.6Hz   40.8bps   Magnetic Variation                      
#     817517  35.4   5.8Hz  407.6bps   Position, Rapid Update                  
#     326996  14.2   2.3Hz  163.0bps   COG & SOG, Rapid Update                 
#      81741   3.5   0.6Hz  222.4bps   GNSS Position Data                      
#      81790   3.5   0.6Hz   40.8bps   Cross Track Error                       
#      81791   3.5   0.6Hz  162.0bps   Navigation Data                         
#      81781   3.5   0.6Hz   40.8bps   GNSS DOPs                               
#      81780   3.5   0.6Hz  707.8bps   GNSS Sats in View                       
#
# Device "VHF 300i AIS" (7359 messages, 61638 bytes, ~3.8bps)
#      Count     %    Freq   Bitrate   Name
#       7323  99.5   0.1Hz    3.7bps   ISO Address Claim                       
#          1   0.0   0.0Hz    0.0bps   Manufacturer Proprietary C              
#         20   0.3   0.0Hz    0.2bps   Product Information                     
#         14   0.2   0.0Hz    0.0bps   Radio Frequency/Mode/Power              
#          1   0.0   0.0Hz    0.0bps   DSC Call Information                    
#
# Device "XB8000F" (601452 messages, 21430313 bytes, ~1246.9bps)
#      Count     %    Freq   Bitrate   Name
#         32   0.0   0.0Hz    0.0bps   ISO Address Claim                       
#         19   0.0   0.0Hz    0.1bps   Product Information                     
#     164856  27.4   1.2Hz   82.2bps   Position, Rapid Update                  
#     164828  27.4   1.2Hz   82.2bps   COG & SOG, Rapid Update                 
#      81127  13.5   0.6Hz  220.8bps   GNSS Position Data                      
#      80414  13.4   0.6Hz  131.7bps   AIS Class A Position Report             
#      27877   4.6   0.2Hz   44.1bps   AIS Class B Position Report             
#          6   0.0   0.0Hz    0.0bps   Cross Track Error                       
#          6   0.0   0.0Hz    0.0bps   Navigation Data                         
#      78678  13.1   0.6Hz  672.5bps   GNSS Sats in View                       
#       2452   0.4   0.0Hz   10.7bps   AIS Class A Static and Voyage Related Data
#         72   0.0   0.0Hz    0.1bps   AIS SAR Aircraft Position Report        
#        319   0.1   0.0Hz    0.5bps   AIS Class B static Part A)              
#        319   0.1   0.0Hz    0.6bps   AIS Class B static Part B)              
#
# Device "WiFi REPLICATOR" (8224 messages, 71840 bytes, ~4.4bps)
#      Count     %    Freq   Bitrate   Name
#       6794  82.6   0.0Hz    3.4bps   ISO Address Claim                       
#         48   0.6   0.0Hz    0.4bps   Product Information                     
#
# Device "DST200" (455886 messages, 4146880 bytes, ~255.8bps)
#      Count     %    Freq   Bitrate   Name
#         33   0.0   0.0Hz    0.0bps   ISO Address Claim                       
#          1   0.0   0.0Hz    0.0bps   NMEA Group Function                     
#          1   0.0   0.0Hz    0.0bps   Manufacturer Proprietary C              
#         20   0.0   0.0Hz    0.2bps   Product Information                     
#      82881  18.2   0.6Hz   41.3bps   Speed                                   
#      82879  18.2   0.6Hz   41.3bps   Water Depth                             
#      82879  18.2   0.6Hz   69.7bps   Distance Log                            
#     165751  36.4   1.2Hz   82.6bps   Environmental Parameters 1              
#          3   0.0   0.0Hz    0.0bps   Airmar POST                             
#
# Device "WX Series Weathe" (4057706 messages, 48765699 bytes, ~2952.4bps)
#      Count     %    Freq   Bitrate   Name
#         33   0.0   0.0Hz    0.0bps   ISO Address Claim                       
#     320194   7.9   2.3Hz  159.7bps   NMEA Group Function                     
#      82868   2.0   0.6Hz   41.3bps   System Time                             
#         19   0.0   0.0Hz    0.1bps   Product Information                     
#     828619  20.4   5.9Hz  413.2bps   Vessel Heading                          
#     828618  20.4   5.9Hz  413.2bps   Rate of Turn                            
#      82869   2.0   0.6Hz   41.3bps   Attitude                                
#      82868   2.0   0.6Hz   41.3bps   Magnetic Variation                      
#     414308  10.2   3.0Hz  206.6bps   Position, Rapid Update                  
#     414307  10.2   3.0Hz  206.6bps   COG & SOG, Rapid Update                 
#      82850   2.0   0.6Hz  206.6bps   GNSS Position Data                      
#      82867   2.0   0.6Hz   41.3bps   Time & Date                             
#       8290   0.2   0.1Hz    9.8bps   Datum                                   
#      82869   2.0   0.6Hz   41.3bps   GNSS DOPs                               
#      82797   2.0   0.6Hz  695.5bps   GNSS Sats in View                       
#     331451   8.2   2.4Hz  165.3bps   Wind Data                               
#     165729   4.1   1.2Hz   82.6bps   Environmental Parameters 1              
#      41434   1.0   0.3Hz   20.7bps   Temperature                             
#      41435   1.0   0.3Hz   20.7bps   Actual Pressure                         
#      82852   2.0   0.6Hz  145.2bps   Meteorological Station Data             
#          6   0.0   0.0Hz    0.0bps   Airmar POST                             

Many message frequencies seem lower than expected.  This could be due to messages being "stomped on" by other devices.  Action for further investigation.

Plots

Finally the script generates a set of plots.  Each plot presents two pie charts - one of message counts and the other of message bytes.  The first plot shows the relative contribution of the devices.  The rest show the messages of each device individually...

############################################################

# Plots

############################################################

#

# Total

# Device "SPx5  E12197"

# Device "GPSmap 451 AUS"

# Device "VHF 300i AIS"

# Device "XB8000F"

# Device "WiFi REPLICATOR"

# Device "DST200"

# Device "WX Series Weathe"

#

############################################################

#                    SaiLog_Stat done                      #

############################################################

Here are the plots, with each followed by comments...

Total Traffic Plot

The first surprise above is the volume of messages from the SPx5 autopilot.  Checking the text output shows these are Rudder messages.  The second pie chart shows that these messages are thankfully short.  The Rudder information isn't used so an action is to investigate disabling these messages.

The second surprise above is the volume of messages from the GPS ChartPlotter.  The second pie chart shows that these messages are long.  The device needs to receive AIS data from the network but its transmitted messages aren't used.  Action is to try to disable its transmissions.

The large message number and data volume from the Weather Station is expected due to the wide variety of data it provides.

Autopilot Traffic Plot

The Autopilot transmits frequent Rudder messages!

GPS ChartPlotter Traffic Plot

The GPS ChartPlotter transmits an unexpectedly varied set of data.  An interesting message type is Sats In View.  The pie charts show they produce high data volume despite being transmitted at a low rate.  ie Each message is long.  Also interesting is the volume of proprietary messages.

VHF Traffic Plot

The VHF transmits rarely.  Its traffic is dominated by ISO Address Claim messages.  Interestingly a single DSC Call Information message occurred.

AIS Traffic Plot

The AIS transmits a variety of information including GPS and AIS data.  Again, the Sats In View message data volume is high.  Contrary to expectation the volume of AIS messages was modest, despite our passage being in the English Channel - one of the busiest seaways in the world.

WiFi Bridge Traffic Plot

The WiFi Bridge transmits rarely.  Its traffic is dominated by ISO Address Claim messages.

Speed/Depth Sensor Traffic Plot

The Speed/Depth Sensor transmits speed, depth, log and temperature data.  Interestingly the water temperature data is transmitted at twice the rate of the other data.

Weather Station Traffic Plot

The Weather Station transmits a wide variety of data including GPS, attitude, attitude rate, wind, temperature and pressure.  Once again the Sats In View message data volume is high.


Overall SaiLog_Stat has provided lots of useful information about our NMEA-2000 network.  It has made plain exactly what is on the bus, how often, and its volume.  Plenty to think about!

Keep well everyone!

SaiLog YDVR NMEA-2000 Data Plotting

Hi everyone,
We've been keeping ourselves busy in isolation aboard Zen Again here in Gosport Premier Marina.  The topic of this post is how we plot the data we record on our NMEA-2000 network.  Yes, back to technical posts!

Zen Again's New Home

Voyage Recorder

Last year we added a Yacht Devices Voyage Recorder (YDVR) to our network.  The device is small with a microSD card plugged in the the end of the unit.  An LED shows whether data is being written.  The unit can also record audio, with a jack at the end of the unit.

YDVR - NMEA-2000 connector indicates its size

The YDVR records all NMEA-2000 data on the network.  On Zen Again this includes the following devices and data:

• Vesper XB8000 AIS
• GPS
• AIS
• Garmin GPSmap451 Chart Plotter
• GPS
• Garmin VHF300iAIS VHF Transceiver
• AIS
• DSC
• Airmar DST800 Speed/Depth Transducer
• Depth
• Speed
• Log
• Airmar 220WX Weather Station
• GPS
• 3D Attitude
• 3D Attitude Rate
• Air Temperature
• Air Pressure

As the above list shows, there's a lot of data.  We have three sources of GPS information - currently at 1Hz from the GPSmap, 2Hz from the Vesper and 4Hz from the 220WX.  AIS is another source of substantial data.  We are using a 32GB microSD card.

The YVDR is configurable to...

• enable/disable NMEA-2000 recording (ie for audio only)
• DAT/CAN file format
• AIS on/off (AIS generates a lot of data)
• ignore nominated NMEA-2000 PGNs (to ignore particular data types)
• ignore address (to ignore particular devices)
• enable/disable audio
• adjust audio settings
• & more

The YDVR comes with Windows and Mac OS X apps called YVDRconv.  The app translates the microSD data files into a variety of formats.  We convert to CanBoat/SignalK log file format which looks like this...

YVDRconv Output Format snippet

Zen Again's log files consume about 20MB/hour.  For our recent passage from London to Gosport the log file is about 800MB.  Given we were in a busy area much of this data may be AIS.

Having recorded the data the question was what can we do with the data?  Our goal was to produce plots showing boat performance and the environment in which we were sailing.  Searching the internet didn't produce anything which looked ideal, although there must be SignalK apps for this somewhere.

SaiLog Plot

We developed a Python script to add to our SaiLog python suite.  Example plots below show our passage from London to Gosport (click on each for full detail)...

London to Gosport Passage - Environment Plot 

The environment plot's wind sub-plots clearly show the nice SW wind we enjoyed on day 1, the light winds overnight and the slowly building NE wind on day 2 which was very gusty.  The water and air temperatures show it was indeed chilly!

London to Gosport Passage - Performance Plot

The performance plot's speed sub-plot clearly shows the effect of currents during the passage.  The heel angle sub-plot shows when seas were present.

SaiLog Plot currently plots selected data from our DST800 and 220WX devices.  All NMEA-2000 data is marked with its source address, which allows us to select the data.

Using YVDRconv

Using YVDRconv is quite simple.  On opening the app there's a welcome page for the conversion wizard...

YVDRconv page 1

The trickiest step is selecting the files to import on the next page.  The YVDR splits files into a time series of folders.  We had to select files from several folders, using the dates of each file to ensure we had all data from our passage time window...

YVDRconv page 2

On page 3 the output format is selected.  We use SignalK log format since we hope to be using SignalK soon, and the format is readily parsed in Python scripts.

YVDRconv page 3

On page 4 the output file is nominated together with various options...

YVDRconv page 4

On page 5 the selections are summarised...

YVDRconv page 5

Clicking Next starts the conversion.  Our 40 hour recording took about 10 minutes to produce an 830MB file.

YVDRconv page 6

On completion you have a number of options.  Take care if the output file is large!  I suggest the Finish button.

YVDRconv page 7

We expect to continue working on SaiLog Plotter to improve its capabilities.

Keep Well Everyone!

London to Gosport

Hi everyone,
For more than nine months we've been planning to spend a year based in Gosport from this April.  With the onset of the coronavirus pandemic we thought hard about whether to stay on in London or proceed.   We decided to proceed.  There are few better places to isolate ourselves than on our wonderful boat!

Our original hope had been to cruise to Gosport over a week or so.  Given the pandemic situation it became a delivery passage.  We departed St Katharine's Dock at 0800 on Wednesday and arrived in Gosport at 2100 on Thursday. Here's our track...

Zen Again Track

And here are the usual statistics...

• Distances/Speeds
• Route Distance = 193nm
• GPS Distance = 194nm!
• Duration = 37 hours
• Average ground speed = 5.0 kt
• Weather
• Minimum wind speed = 0 kt
• Average wind speed = 10 kt
• Maximum wind speed = 16 kt
• Apparent wind angle range = 45 to 170
• Seas up to 1m
• Partly cloudy on Wednesday.  Overcast on Thursday with drizzle in fog.
• Engine
• Total = 24 hours
• Driving = 24 hours
• Charging = 0 hours

We were up and ready to go at 0730.  We departed SKD at 0800 and high tide was at about 0830.  On exit we gave Zen Again a last look at Tower Bridge before setting course down river.

Entering the SKD lock

Outer lock gate going down

Zen Again paying her respects to Tower Bridge

Through the Thames Barrier

We had a nice SW breeze of 10-15 knots to take us out of the Thames Estuary.  We set sail at Tilbury where the river's meanders lessen.  The clouds were slowly dispersing.

Sailing!

We sailed close past the Red Sands Sea Forts which I used to race to with my uncle and cousin in the 1980s.

Red Sands See Forts (WW2 Anti-Aircraft Platforms) 

A great sail out of the Thames Estuary

Our new kit performed very well.  The wind data from our Airmar 220WX seemed accurate.  SaiLog displayed the data nicely.  And iNavX worked nicely on the iPads in their Armor-X cases and mounts.

iPad Armor-X mount and case working well

By Wednesday evening we were around North Foreland and approaching Ramsgate.  We'd had a fantastic sail all day, some of it in glorious sunshine.  The breeze was cold though.

Passing Ramsgate the wind fell very light and the engine went on.  Sadly it didn't go off until we arrived in Gosport Marina.  We motor-sailed for several hours until the wind was too light.  A weak warm front was passing over us, bringing a NE wind perfect for our passage west.  Sadly it didn't arrive until Thursday afternoon.

The night was overcast and cold - down to about 6C according to the 220WX.   Many layers, thick socks, scarves and hats were applied.  We discovered ugg boots are excellent dry-weather winter sea boots - toasty warm!  The traffic in and out of Dover seemed as busy as ever.

Wednesday morning delivered a low overcast which descended to become fog for several hours.  Visibility was less than 1/2nm.  Our nav lights stayed on all day.

Fog along the south coast

In the afternoon a flock of small birds decided to circle us.  At least four landed aboard and scoured the boat for insects.  One of them seemed very weak and stayed with us.  One of its pals came back and seemed to console it but then flew off.  Sadly our little pal passed away and we buried it as sea.

Stray bird and Ugg boots

We entered the Solent at twilight.  There was a steady stream of large commercial vessels entering the Solent.  We kept outside the main channels and communicated with Southampton VTS and the Queen's Harbour Master (QHM).  Small craft enter and leave Portsmouth using the far western side of the channel.

Route into Portsmouth Harbour and Gosport Marina

Arriving in an unfamiliar marina at night is always interesting, especially in tidal areas.  GPS certainly helps.  The marina staff gave clear directions as we crept into the marina dead-slow and berthed without incident.

We're currently in the far NE corner of the marina.  From our berth we can see the nightly light show on the Spinnaker Tower.  We can also see the RN's new aircraft carrier HMS Queen Elizabeth.

In Gosport we hope we'll be able to go sailing frequently - if only for day sails in the Solent initially.

This afternoon we checked out the local supermarkets.  Like those in London there were many bare shelves.  We always keep Zen Again well stocked but hopefully the panic buying will settle down soon.

Trust all's well where you are!