Generic Support for Session and Lap in FIT files

GENERIC SUPPORT FOR PARSING INTO XDATA

.. Generically parse FIT file messages into XDATA. The current
   implementation does this for session, lap and totals messages
   but could very easily be extended to any other message type

.. Generic parsing uses metadata rather than hard coding the
   message and field types and so on

.. The FIT metadata (FITmetadata.json) has been expanded to
   include definitions of message types and all the standard
   fields within the message types

.. The existing hard-coded parsing remains to extract data
   and apply directly to ridefile samples and metadata. The
   generic parser simply adds additional tabs on the data
   view as XDATA so users can access it.

   CODE REFACTORING, COMMENTS AND BUG FIXES

.. At some point the code needs to be refactored as it is
   janky and needs to align with the rest of the codebase

.. Includes a mild refactor renaming some of the classes/structs
   and variables to reflect what they actually are, for example:

       FitFileReadState -> FitFileParser
       FitDefinition -> FitMessage

.. Added lots of code comments and re-organised the code
   into clear sections to help navigate what is a very
   cumbersome source file, this breaks git blame history
   but is worth the loss (you can checkout an earlier commit
   to do a full blame)

.. Changed debugging levels to be more helpful

.. Generally I did not change any code, but there were a
   couple of serious bugs that needed to be corrected:

      Field definitions gets the type wrong in a couple of
      places since the type is stored in the low 4 bits:
      type = value & 0x1F

      The decodeDeveloperFieldDescription function did not
      check for NA_VALUEs for scale, offset, native field

.. For less serious bugs I added FIXME comments throughout the code

Fixes #4416

src/Core/GcUpgrade.cpp

@@ -406,6 +406,17 @@ GcUpgrade::upgrade(const QDir &home)
     //----------------------------------------------------------------------
 
 
+    //----------------------------------------------------------------------
+    // Current development release processing
+    //----------------------------------------------------------------------
+    if (last < VERSION37_BUILD) {
+        // remove FITmetadata.json since it now has message number metadata
+        // any previously cached version would ordinarily override the internal
+        // version, and if there is no internet it will never get refreshed
+        // so we delete the cache to ensure the built-in (current latest) is used
+        QString filename = home.canonicalPath()+"/../FITmetadata.json";
+        QFile::remove(filename);
+    }
 
     //----------------------------------------------------------------------
     // All Version dependent Upgrade Steps are done ...

src/Core/GcUpgrade.h

@@ -106,7 +106,8 @@
 // 4011 - V3.6 DEVELOPMENT 2208 (AUG 2022) (Pre-release RC2)
 // 4012 - V3.6 DEVELOPMENT 2210 (OCT 2022) (Pre-release RC3)
 // 4013 - V3.6 DEVELOPMENT 2303 (MAR 2023) (Pre-release RC4)
-// 5000 - V3.6 RELEASE (August 2023) - latest snapshot 4/11
+// 5000 - V3.6 RELEASE (August 2023) - latest snapshot 24/9
+// 5001 - V3.7 DEVELOPMENT (NOV 2023)
 
 #define VERSION3_BUILD    3010 // released
 #define VERSION3_SP1      3030 // released
@@ -118,6 +119,7 @@
 #define VERSION34_BUILD   3955 // released
 #define VERSION35_BUILD   3990 // released
 #define VERSION36_BUILD   5000 // released 5/8/23
+#define VERSION37_BUILD   5001 // released 5/8/23
 
 // will keep changing during testing and before final release
 #define VERSION31_BUILD VERSION31_UPG

src/FileIO/FitRideFile.h

@@ -32,5 +32,89 @@ struct FitFileReader : public RideFileReader {
 
 };
 
+#ifndef MATHCONST_PI
+#define MATHCONST_PI            3.141592653589793238462643383279502884L /* pi */
+#endif
+
+#define DEFINITION_MSG_HEADER   64
+#define FILE_ID_MSG_NUM         0
+#define SESSION_MSG_NUM         18
+#define LAP_MSG_NUM             19
+#define RECORD_MSG_NUM          20
+#define EVENT_MSG_NUM           21
+#define ACTIVITY_MSG_NUM        34
+#define FILE_CREATOR_MSG_NUM    49
+#define HRV_MSG_NUM             78
+#define SEGMENT_MSG_NUM         142
+
+/* FIT has uint32 as largest integer type. So qint64 is large enough to
+ * store all integer types - no matter if they're signed or not */
+
+// this will need to change if float or other non-integer values are
+// introduced into the file format *FIXME*
+typedef qint64 fit_value_t;
+
+#define NA_VALUE std::numeric_limits<fit_value_t>::max()
+#define NA_VALUEF  (double)(0xFFFFFFFF)
+
+typedef std::string fit_string_value;
+typedef float fit_float_value;
+
+struct FitField {
+    int num;
+    int type; // FIT base_type
+    int size; // in bytes
+    int deve_idx; // Developer Data Index
+};
+
+struct FitFieldDefinition {
+    int dev_id; // Developer Data Index (for developer fields)
+    int num;
+    int type; // FIT base_type
+    bool istime; // for date_time fields with base type uint32
+    int size; // in bytes
+    int native; // native field number
+    double scale; // should be a double not an int
+    int offset;
+    fit_string_value message; // what message section does this belong to
+    fit_string_value name; // what is the name of the field
+    fit_string_value unit; // what units does the field use
+};
+
+struct FitDeveApp {
+    fit_string_value dev_id; // Developer Data Index
+    fit_string_value app_id; // application_id
+    int man_id; // manufacturer_id
+    int dev_data_id; // developer_data_index
+    qint64 app_version; // application_version
+
+    QList<FitFieldDefinition> fields;
+};
+
+struct FitMessage {
+    int global_msg_num;
+    int local_msg_num;
+    bool is_big_endian;
+    std::vector<FitField> fields;
+};
+
+enum fitValueType { SingleValue, ListValue, FloatValue, StringValue };
+typedef enum fitValueType FitValueType;
+
+struct FitValue
+{
+    FitValueType type;
+    fit_value_t v;
+    fit_string_value s;
+    fit_float_value f;
+    QList<fit_value_t> list;
+    int size;
+};
+
+// Fit types metadata
+struct FITproduct { int manu, prod; QString name; };
+struct FITmanufacturer { int manu; QString name; };
+struct FITmessage { int num; QString desc; };
+
 #endif // _FitRideFile_h
 

src/FileIO/RideFile.h

@@ -189,7 +189,7 @@ class RideFile : public QObject // QObject to emit signals
         // fix tools
         friend class FixLapSwim;
         friend class Snippets;
-        friend struct FitFileReaderState;
+        friend struct FitFileParser;
 
         // utility
         static unsigned int computeFileCRC(QString); 
@@ -551,7 +551,7 @@ class RideFileIterator {
         int start, stop, index;
 };
 
-#define XDATA_MAXVALUES 32
+#define XDATA_MAXVALUES 64
 
 class XDataPoint {
 public:

util/fit/Makefile

@@ -1,3 +1,3 @@
-../../src/Resources/json/FITmetadata.json: fit_example.h nongarmin.json fitprod.py
+../../src/Resources/json/FITmetadata.json: fit_example.h nongarmin.json fields.json fitprod.py
 	python3 fitprod.py > ../../src/Resources/json/FITmetadata.json
 

util/fit/README

@@ -3,12 +3,36 @@ FITmetadata.json generator
 
 For Garmin device updates:
 
-    Make sure you download the latest FIT SDK from thisisant.com and extract
+    Make sure you download the latest FIT SDK from https://developer.garmin.com/fit/download/
-    the c/fit_example.h file and place it in this directory.
+    and extract the c/fit_example.h file and place it in this directory.
 
     Do not edit this file, we expect it to be lifted directly from the SDK
     and it will be replaced as new versions of the SDK are released
 
+For Garmin profile updates:
+
+   The fields.json file in this directory is metadata describing most of the standard
+   fields and is based upon the Profile.xls file in the SDK. If new fields are or profiles
+   are added and you want them to be parsed via metadata then update fields.json to
+   include the entries.
+
+   At present the FIT ride file reader code is a mix of hard coded parsing for
+   the majority of the message types, but the code for parsing session and lap
+   message types does use the metadata to control parsing (and puts the results
+   into the XDATA section of a ridefile)
+
+   If you add new profile then you should use metadata to drive the parsing and
+   add the data to the XDATA section to avoid impacting existing code.
+
+Generate a new FITmetadata.json file
+
+    There is a Makefile in this directory that will refresh baed upon the timestamp
+    of the fit_example.h. To refresh simply run 'make'. If you wish to re-run and
+    regenerate 'touch fit_example.h; make' will do that.
+
+    You will need python3 installed since the generator is a python script. Please
+    do not edit this script but raise a github issue if you find it is broken.
+
 When Garmin devices are not yet supported by the SDK
 
     The SDK lags behind device availability in this case you should add

util/fit/fitprod.py

@@ -5,13 +5,17 @@ print('{\n    "VERSION":' + str(int(time.time())) + ',')
 print('    "COMMENT":"Do not edit this file directly it is generated.",')
 print('    "PRODUCTS":[')
 
+#
 # output the known products captured by GoldenCheetah users
+#
 nongarmin = open("nongarmin.json", "r")
 lines = nongarmin.readlines()
 for line in lines:
     print("        " + line, end="")
 
+#
 # output garmin products as described in the FIT SDK
+#
 sdkheader = open("fit_example.h","r")
 lines = sdkheader.readlines()
 pre="        "
@@ -26,8 +30,9 @@ for line in lines:
             pre=",\n        "
 print("\n    ],\n")
 
+#
 # manufacturers list from FIT SDK
-
+#
 print('    "MANUFACTURERS":[')
 pre="        "
 for line in lines:
@@ -36,5 +41,29 @@ for line in lines:
         print(pre+ '{ "manu":' + match.group(2).strip() + ', "name":"' +  match.group(1).strip().replace('_',' ').title() + '" }', end="")
         pre=",\n        "
 
-print("\n    ]\n}")
+print("\n    ],\n")
 
+#
+# field numbers
+#
+fields = open("fields.json", "r")
+fieldlines = fields.readlines()
+for line in fieldlines:
+    print(line, end="")
+
+#
+# Message number description
+#
+# // #define FIT_MESG_NUM_HR_ZONE                                                     ((FIT_MESG_NUM)8)
+
+print('    "MESSAGES":[')
+pre="        "
+for line in lines:
+    match = re.search("FIT_MESG_NUM_([^ \t]*).*\(\(FIT_MESG_NUM\)([ 0-9]*)", line)
+    if match:
+        num = int(match.group(2).strip())
+        if num > 0:
+            print(pre+ '{ "num":', num, ', "desc":"' +  match.group(1).strip().replace('_',' ').title() + '" }', end="")
+            pre=",\n        "
+
+print("\n    ]\n}")