The RIM BlackBerry Serial Protocol

Phil Schwan <phil@off.net>
Mike Shaver <shaver@off.net>
Ian Goldberg <ian@zeroknowledge.com>

26 August 2001
Draft 0.3

Table of Contents

1.0Introduction
1.1Known Deficiencies
1.2Protocol Overview
1.3Packet Descriptions
1.4Normal Command Packets
1.5Extended Packets
1.6ACK Packets
1.7Date and Time Encoding
 
2.0Initialization
2.1Handheld Information Request
2.2Command Table
 
3.0Database Access
3.1Database Database
3.2Record Count
3.3Retrieve Database
3.4Download Record
3.5Clear Database
3.6Upload Record
3.7Set Time
3.8Example Database Backup
3.8Example Database Restore
 
4.0Database Formats
4.1AutoText
4.2Ribbon Positions
4.3Memos
4.4Calendar
4.5Address Book
 
5.0Packet Checksum
 
6.0Acknowledgements
 
7.0Changelog

1.0 Introduction

The information contained herein was acquired without the help, or indeed the blessing, of Research In Motion®, the manufacturer of the BlackBerry™. Requests for this information were denied, as were requests for the source code to various applications and operating system libraries.

It is our hope that RIM® will understand that there is a community of paid and unpaid developers who are willing to spend their time generating high-quality software that runs on the BlackBerry, and that it is in only their best interests to make that process as easy as possible. We hope that RIM will take this opportunity to correct inaccuracies or ignorances in this document, and to work with us or others to increase the breadth of available information regarding the internals of the BlackBerry.

Thus it should come as no surprise to learn that any inaccuracies or omissions are entirely our fault, not those of Research In Motion. This document is provided under the OpenContent License version 1.0 and is provided without any warranty, express or implied.

1.1 Known Deficiencies

Although this is a work-in-progress, I decided that it was better to release early, release often, simply because I don't have as much time to dedicate to it as I'd like. This is my list of known bugs; it is almost certainly not comprehensive:

1.2 Protocol Overview

For the remainder of the document, it is understood that we are describing the serial protocol which is used to backup, restore, and synchronize data between the BlackBerry handheld unit and the desktop software. Other protocols, such as the Radio Access Protocol, which may also use the serial line are outside of the scope of this document.

When the BlackBerry and desktop software communicate, they do so with a protocol made up of simple packets and single byte return codes. All packets have the same basic structure:

BytesDescription
3Packet header

Always D9 AE FB
1Command type

Each comand type has a unique value, which will limit the set of commands available:
1Command

For "Command Type" 41, see section 1.6

For "Command Type" 40, the value 00 specifies initialization-related commands. Any other value represents commands listed in the "Command Table". See section 2.2 for more information regarding the command table.

For "Command Type" 60, the only observed value has been 02.
variableCommand-dependent packet data
3Footer

Always BF EA 9D

1.3 Packet Descriptions

For the remainder of this document:

1.4 Normal Command packets

For normal command packets, the data is of the format:

BytesDescription
2Length (MSB)

The length of the data to follow.
"Length"Payload

The structure and meaning of the rest of the packet depends on the Command Type.
2Checksum

A checksum of the payload. See Section 5.0 for the checksum description.

1.5 Extended Packets

When a packet would otherwise exceed 1024 bytes, it is instead split into a number of 1024 byte Extended packets followed by a standard packet of less than 1024 bytes. Although one assumes that any packet of more than 1024 bytes would be split in this fashion, the only observed packets of this size have been Normal Command packets.

Extended packets share the same structure as normal command packets:

BytesDescription
2Length (MSB)

The length of the data to follow.
"Length"Payload

See below.
2Checksum

A checksum of the payload. See Section 5.0 for the checksum description.

For example, when the handheld sends a large database record (say, 3000 bytes) to the desktop software, it would send packets of the following form:

Extended packet, containing bytes 0-1023
Extended packet, containing bytes 1024-2047
Normal Command packet, containing bytes 2048-2999

On the wire:

Command TypeCommandLength PayloadChecksum
600204 001024 bytes93 78
600204 001024 bytes8A 92
400203 B8952 bytesFF 1C

All three packet payloads are concatenated and treated as a single very large Normal Packet by the parser in the desktop software.

The desktop software sends a single byte 0x06 to acknowledge the successful receipt of an Extended packet. The presence of Extended packets does not affect the treatment of or reply to the Normal Command packet that follows.

1.6 ACK Packets

For ACK packets, the data is of the format:

BytesDescription
2Status code

Observed values are:
  • 00 00 = Success
  • 02 00 = Too few arguments
  • 05 00 = Unknown subcommand [unconfirmed]

1.7 Date and Time Encoding

Whenever dates and times are used, with few exceptions, they are encoded in the very same way:

BytesDescription
1Seconds
1Minutes
1Hours
1Day
1Month

The month of the year, starting with January == 1
1Day of the Week

The day of the week, starting with Sunday == 0
2Year (LSB)
1Time Zone

GMT == 0; All other zones are encoded as their GMT-delta, counted in half-hour increments. ie, EST == GMT - 5, which would be encoded as -10.

2.0 Initialization

The BlackBerry will only recognize the starting sequence of bytes for short period of time after DTR is reset. When the desktop software notices that the BlackBerry device has been inserted into its cradle, it sends the handshake challenge (header + 0xCF + footer).

The BlackBerry responds with the handshake reply without a footer--the only observed instance of a packet without a footer.

Following that handshake, a number of normal command packets will exchanged to establish the communication parameters and retrieve basic information.

2.1 Handheld Information Request

To retrieve basic information about the handheld, the desktop software sends an Initialization packet with a one-byte (0x00) payload. On the wire:

Command TypeCommandLength PayloadChecksum
400000 0100D8 8C

It receives from the BlackBerry:

BytesDescription
5Unknown data

Unknown purpose.
4MAN number (MSB)

The Mobitex Access Number, or MAN number, is the BlackBerry's unique device number, used to identify itself on the Mobitex network. For more information, see the RIM BlackBerry Radio API or the Mobitex Interface Specification.
4Baud rate (MSB)

The maximum supported baud rate, perhaps?
2Unknown data

Unknown purpose

On the wire:

Command TypeCommandLength UnknownMAN NumberBaud Rate UnknownChecksum
400000 0F00 01 00 00 01 00 46 48 FB00 01 C2 0004 00 94 6A

2.2 Command Table

Presumably to allow for an arbitrarily extensible protocol, the allowed commands list is stored in a special table that the desktop software downloads as a part of its initialization sequence.

The command table request packet is of the format:

BytesDescription
4Baud rate (MSB)

The baud rate at which the remainder of this session will take place.
2Unknown data

Unknown purpose

on the wire:

Command TypeCommandLength Baud RateUnknownChecksum
400000 06 00 00 25 8008 00 0A 34

The BlackBerry responds with a standard command/length packet beginning, followed by records of the format:

BytesDescription
1Name Length

The length of the name of the database, in bytes. This is one of the very few string lengths which does not include a trailing nul.
1Command Number

The numeric command identifier.
"Name Length"Command Name

The text command identifier, without trailing nul.

on the wire:

Command TypeCommandLength Name LengthCommand NumberCommand Name Checksum
404C00 50
0E01Device Options
0F02 Database Access
0B03Random Seed
0C04Service Book
1205Host Routing Table
00 00

3.0 Database Access

The majority of the interesting things that are likely to take place between your unit and your desktop involve database operations: backups, restores, hotsyncing, etc. The BlackBerry makes use of a number of internal databases as well, for which you'll need at least a cursory understanding of the database access protocol:

BytesDescription
1Database Instruction

Defines the operation to take place on the given database. Currently known database instructions are:
variableAdditional Data

The structure and meaning of the rest of the packet depends on the Database Instruction.

From the sample command table in section 2.2, we learn that for this device, the command for database access is 02. Thus, the beginning of any database command or reply on this device is easily recognizable by the "40 02" after the transmission header.

3.1 Database Database

The internal list of existing databases is queried during the initial handshake process after the command table, and provides the information needed to carry out many of the rest of the database operations.

The command to retrieve the database database is a simple one byte packet that contains just the Database Instruction 4C.

On the wire:

Command TypeCommandLength DB InstructionChecksum
400200 01 4C4C C6

In reply, the BlackBerry provides the database database:

BytesDescription
1Database Instruction

For "Database Database", 4C
2Record Count (LSB)

The number of records in the database database (ie, the number of databases in the unit.)

and for each record of the database database:

BytesDescription
2Database Number

Numeric database identifier. Used to reference this database in future database operations.
1Unknown data

Unknown purpose. Always observed to be 00.
4Database Size (LSB)

The size of the database, in bytes.
2Record Count

The number of records in this database.
2Unknown data

Unknown purpose.
2Database Name Length (LSB)

The length of the name of the database, including trailing nul.
"Name Length"Database Name

The text name of the database, including trailing nul.

On the wire:

Command TypeCommandLength DB InstructionDatabase Count DB NumberUnknownDatabase Size Record CountUnknownName Length Database NameChecksum
400202 454C16 00
01 00008E 0F 00 00 60 0000 0009 00Autotext 00
02 000006 04 00 00 10 0000 0011 00Ribbon Positions 00
03 000000 00 00 00 00 0000 0007 00Policy 00
04 000087 05 00 00 0A 0000 0006 00Memos 00
05 000070 00 00 00 02 0000 0013 00Host Routing Table 00
06 0000BF 02 00 00 09 0000 0006 00Tasks 00
07 000011 22 00 00 48 0000 D717 00Calendar 00
08 0000D7 17 00 00 45 0000 000D 00Address Book 00
09 000086 00 00 00 01 0000 000D 00Service Book 00
0A 000087 37 00 00 07 0200 0009 00Messages 00
0B 000046 42 00 00 F3 0100 0010 00Purged Messages 00
0C 000000 00 00 00 00 0000 0008 00Folders 00
0D 000044 02 00 00 02 0000 0009 00Searches 00
0E 000000 00 00 00 01 0000 0015 00Address Book Options 00
0F 000000 00 00 00 01 0000 000E 00Email Service 00
10 000000 00 00 00 01 0000 000A 00Folder Id 00
11 000000 00 00 00 01 0000 0015 00Message List Options 00
12 000000 00 00 00 01 0000 0010 00DefaultServices 00
13 000000 00 00 00 01 0000 000F 00Device Options 00
14 000000 00 00 00 01 0000 0011 00Calendar Options 00
15 000000 00 00 00 01 0000 0017 00Calendar Sync Bookmark 00
16 000000 00 00 00 01 0000 000E 00Tasks Options 00
56 4E

3.2 Record Count

When the desktop software wants to know how many records are in a given DB, it issues the "record count" instruction:

BytesDescription
1Database Instruction

For "Record Count", 4E
2Database ID (LSB)

The numeric database ID to query. See Database Database.

On the wire:

Command TypeCommandLength DB InstructionDB Number Checksum
400200 034E0D 00ED 47

and receives in reply the single byte 0x06 followed by:

BytesDescription
1Database Instruction

For "Record Count", 4E
4Record Count (LSB)

The number of records in the queried database.

On the wire:

Command TypeCommandLength DB InstructionRecord Count Checksum
400200 054E03 00 00 00 A5 D6

3.3 Retrieve Database

When the desktop software is ready to backup or sync any normal database, it will issue the "Retrieve Database" command. This command is identical to the "Record Count" command, only with a different Database Instruction (42).

On the wire:

Command TypeCommandLength DB InstructionDB Number Checksum
400200 034206 00E6 06

3.4 Download Record

In response to a "Retrieve Database" command, the BlackBerry will send a single byte 0x06 followed by a series of "Download Record" packets with a Database Instruction of 0x44.

Databases are composed of records, which are composed of fields. Each record is sent back in its own packet; the structure and meaning of the field payload depends on the database. After each of these record packets, the desktop software sends a single byte 0x06 followed by an ACK packet. When the last record has been sent, the BlackBerry sends an ACK instead of another record.

The record packets are as follows:

BytesDescription
1Database Instruction

For "Download Packet", 44
1Unknown

Always observed to be 00
2Record Handle (LSB)

A unique pointer to this record in the handheld. See page 45 of the RIM Database API Version 2.0 for more information regarding DatabaseRecords.
5Unknown

Unknown purpose
VariableField Data

Described below

Although the purpose of those middle bytes is as-yet-unknown, it's interesting to note that they are stored in the Inter@ctive Pager Backup file and re-sent during a Upload Record operation.

For each field in the record, the following field data is sent:

BytesDescription
2Field Length (LSB)

The length of the field data to follow, not including the length of the field ID.
1Field ID

The not-necessarily-unique field identifier.
"Field Length"Field Payload

Field data

These packets are only ever sent from the BlackBerry to the desktop software--when the desktop sends to the handheld, the "Upload Record" packet is used.

On the wire:

Command TypeCommandLength DB InstructionUnknownRecord Handle Unknown Field LengthField IDField Payload Checksum
400200 41440058 0A87 46 27 07 01
01 000174
07 0002Task 1 00
04 000901 00 00 00
04 000A02 00 00 00
05 0003Note 00
04 000801 00 00 00
0A 000536 2D 01 19 08 06 D1 07 D8 09
FF 11

In response, the desktop software sends a single byte 0x06 followed by an ACK packet.

3.5 Clear Database

At the request of the user--or when the desktop software is about to restore a backed up database--it issues the "Clear Database" instruction. This command is identical to the "Record Count" command, only with a different Database Instruction (0x43).

On the wire:

Command TypeCommandLength DB InstructionDB Number Checksum
400200 034306 003A 5C

In response, the BlackBerry sends a single byte 0x06 followed by an ACK packet.

3.6 Upload Record

In contrast to the stateful nature of the database backup protocol (the desktop software must remember which database is being acted on), the restoration packets have all of the necessary information included:

BytesDescription
1Database Instruction

For "Upload Record", 41
2Database ID (LSB)

The numeric database ID to query. See Database Database.
1Unknown

Always observed to be 00
5Unknown

Unknown purpose
VariableField Data

The structure and meaning of the rest of the packet depends on the Database ID.

Note that the unknown bytes are identical to the unknown bytes received with the record in the Download Record packet.

For each field in the record, the following field data is sent as in the Download Record packet:

BytesDescription
2Field Length (LSB)

The length of the field data to follow, not including the length of the field ID.
1Field ID

The not-necessarily-unique field identifier.
"Field Length"Field Payload

Field data

These packets are only ever sent from the desktop software to the BlackBerry--when the handheld sends to the desktop, the "Download Record" packet is used.

Uploading the sample record from section 3.4 would yield the following on the wire:

Command TypeCommandLength DB InstructionDB NumberUnknown Unknown Field LengthField IDField Payload Checksum
400200 414106 0000 87 46 27 07 01
01 000174
07 0002Task 1 00
04 000901 00 00 00
04 000A02 00 00 00
05 0003Note 00
04 000801 00 00 00
0A 000536 2D 01 19 08 06 D1 07 D8 09
1C 69

In response, the BlackBerry sends a single byte 0x06 followed by an ACK packet.

3.7 Set Time

Although it's not clear why this is a database operation, the set time command follows the regular database command format, and uses the time format described in Section 1.7, Date and Time Encoding.

BytesDescription
1Database Instruction

For "Set Time", 59
9Date and Time

The date and time, as described in Date and Time Encoding.
2Unknown Data

Unknown purpose. Observed to be "00 80".

On the wire:

Command TypeCommandLength DB InstructionDate and TimeUnknown Checksum
400200 0C 5921 1B 03 04 02 01 D1 07 CE00 80 21 1B

3.8 Example Database Backup

Descriptions of the individual packets are helpful, but don't provide a complete picture of a database backup. The following diagram shows what takes place when the desktop software backs up the AutoText database, although the exchange is representative of any typical database backup:

Desktop Software BlackBerry
Record count
  • database 01
Record count
  • 03 records
Return code 06
ACK
ACK
Retrieve database
  • database 01
Return code 06
Download record
  • field 01: arent
  • field 02: aren't
Return code 06
ACK
Download record
  • field 01: cant
  • field 02: can't
Return code 06
ACK
Download record
  • field 01: wont
  • field 02: won't
Return code 06
ACK
ACK

3.9 Example Database Restore

Likewise, this diagram describes the packet flow when the desktop software restores that same database to the handheld:

Desktop Software BlackBerry
Clear Database
  • database 01
Return code 06
ACK
Upload Record
  • database 01
  • field 01: arent
  • field 02: aren't
Return code 06
ACK
Upload Record
  • database 01
  • field 01: cant
  • field 02: can't
Return code 06
ACK
Upload Record
  • database 01
  • field 01: wont
  • field 02: won't
Return code 06
ACK

4.0 Database Formats

Although much of this section is more generally related to the in-flash storage of database records, it's useful to include for the application developer wishing to actually use the data present in the serial backup. It should probably become a completely separate document at some point, perhaps after I've filled in a few of the unknowns.

See Section 3.4 for more information about how records are transferred.

4.1 AutoText

The fields of the AutoText database are simple and pretty much self-explanitory.

FieldDescription
01Match Text

The text to be replaced in an AutoText edit.
02Replacement Text

The text to replace it with.

4.2 Ribbon Positions

Ribbon Positions is a less well-understood database.

FieldDescription
01Application Name
02Ribbon Placement
03Unknown

Almost always observed as FF FF FF FF
04Unknown

Always observed as empty (length 0)
05Unknown

Always observed as empty (length 0)
06Unknown

Always observed as empty (length 0)

Field 02 is the order in which items appear in the ribbon list, but it appears to often be both discontinuous and out of date. For example, my Ribbon Positions database contains ribbon positions for applications which are no longer installed (presumably in case they are ever re-installed), and skips from 0x000A to 0x0016 to 0x03E6. The significance of the high bits and discontinuity, if any, are as-yet-unknown.

4.3 Memos

FieldDescription
01Subject
02Memo Text

4.4 Calendar

FieldDescription
01Unknown
02Subject
03Notes
04Location
05Notification Time
06Start Time
07End Time
12Recurrance Data
16Version Data
26Notification Data

Fields 02, 03, and 04 are standard, nul-terminated strings. Fields 06 and 07 are standard date/time sequences. Field 05 is also a date/time sequence, but with one small twist: if this record is non-recurring and byte 5 is greater than 0x80, then the notification has already passed. This doesn't take place in recurring calendar entries: recurring entries rely solely on field 26.

Field 16 is a 32-bit LSB value which is increased when modifications to the record are made.

Field 26 is a 32-bit LSB value which is the last notification alarm which has been tripped, stored as the number of minutes elapsed since midnight 01 January, 1900. An open question: What happens when, in a non-recurring entry, byte 5 of field 05 disagrees with field 26?

Field 12 is an 18-byte field divided up as follows:

ByteDescription
1Recurrance Type
  • 01 = Day
  • 03 = Month, by date
  • 04 = Month, by day
  • 05 = Year, by date
  • 06 = Year, by day
  • 0C = Week
2Unknown

Always observed as 01
3-4Recurrance Interval (LSB)
5-8Start of Recurrance
9-12End of Recurrance
13-18Additional Data

Depends on byte 1

Bytes 13-18 are further divided up as follows:

Recurrance TypeByteDescription
Day13-18None

Always 00 00 00 00 00 00
Month, by date13The day of the month to recur on.
14-18None

Always 00 00 00 00 00
Month, by day13The day of the week to recur on (Sunday == 0)
14The week of the month on which to recur. Presumably always between 01 and 05, where 05 indicates the last specified day of the week occuring that month.
15-18None

Always 00 00 00 00
Year, by date13The day of the month to recur on
14None

Always 00
15The month of the year to recur on (January == 01)
16-18None

Always 00 00 00
Year, by day13The day of the week to recur on (Sunday == 0)
14The week of the month on which to recur. Presumably always between 01 and 05, where 05 indicates the last specified day of the week occuring that month.
15The month of the year to recur on (January == 01)
16-18None

Always 00 00 00
Week13Days of the week

A bitmap, with each day of the week assigned to the lowest 7 bits (Sunday = 1, Tuesday = 2, Wednesday = 4, etc.)
14-18None

Always 00 00 00 00 00

4.5 Address Book

The field descriptions are helpfully described in the include/AddressFieldType.h file included with the RIM BlackBerry developer kit. It's worth noting that this can never be an exhaustive list, as applications are free to define their own custom field types. This is merely a list of the interesting default fields.

FieldDescription
01Email
02Phone
03Fax
06Work Phone
07Home Phone
08Mobile Phone
09Pager
10PIN
32Name

There are commonly two fields of this type: the first in the record is for the first name, and the second for the last name.
33Company
34Default Communications Method
35Address, Line 1
36Address, Line 2
37Address, Line 3
38City
39State/Province
40Postal Code
41Country
42Title
43Public Key
64Notes
255Invalid Field

5.0 Packet Checksum

As described in section 1.4, each normal command packet contains a checksum at the end. It wasn't defined in any source code or documentation that we could find, and our request for information was met with silence, so I did the only reasonable thing: I gave Ian a few pages of packets and their checksums, and he figured it out. The following Perl code snippet takes packet data and returns the CRC:

sub crc16 ($$) {
    my $crcval = shift;
    my $cval = shift;
    my $poly = 0x8408;

    $crcval ^= $cval;

    my $i;
    for ($i = 0; $i < 8; ++$i) {
        $crcval = $crcval & 1 ? ($crcval >> 1) ^ $poly : $crcval >> 1;
    }
    return $crcval;
}

sub checksum ($$) {
    my $str = shift;
    my $crc = 0xae9e;

    my $i;
    for ($i = 0; $i < length($str); $i++) {
        $crc = crc16($crc, unpack("x${i}C", $str));
    }

    $crc ^= 0x07f0;

    return $crc;
}

6.0 Acknowledgements

RIM and Research In Motion are registered trademarks of Research In Motion Limited. RIM, BlackBerry, and Research In Motion are trademarks of Research In Motion Limited. All other brands, product names and company names mentioned herein may be trademarks or registered trademarks of their respective owners.

7.0 Changelog

Draft 0.3 [26 August 2001]:

Draft 0.2 [25 August 2001]:

Thanks to Joe Shaw for proofreading and providing much helpful feedback.