There are an abundence of information online about how to get OSPF running correctly over WireGuard, however I was not able to find any single post that summarizes everything well. After a bunch of trial and error, I was finally able to get OSPF and OSPFv3 runing over WireGuard tunel successfully. Here are my lessons learned:

The topology is the simplest (addresses are for illustration purpose only):

                                            +--------------------+
                                            |                    |
                             ---------------|       Client       |
                            /  WireGuard    |  192.168.123.2/24  |
                           /                |     fe88::2/64     |
+-----------------------+ /                 |     2000::2/64     |
|                       |/                  |                    |
|        Server         |                   +--------------------+
|    192.168.123.1/24   |
|      2000::1/64       |
|      fe88::1/64       |\
|                       | \
+-----------------------+  \                +--------------------+
                            \  WireGuard    |                    |
                             ---------------|       Client       |
                                            |  192.168.123.3/24  |
                                            |     fe88::2/64     |
                                            |     2000::2/64     |
                                            |                    |
                                            +--------------------+

We would like both clients to establish OSPF neighbor with the server, over the WireGuard tunnel.

OSPF

For OSPF, the config is relatively straightforward. Once both tunnels are up and you can ping server successfully from both clients, use a bird.conf similiar to the following on clients:

protocol ospf {
    area 0.0.0.0 {
        interface "wg0" {
            type ptmp;
            neighbors {
                192.168.123.1;
            };
        };
    };
}

Note the type ptmp config which tells BIRD the connection is Point-to-Multipoint, which tells BIRD that broadcast based neighbot discovery will not work.

Instead, we specify the neighbot (in this case, the server) manually.

Should it work? Unfortunately, after this config, clients still couldn't establish OSPF neighbors with the server, and they are stuck at the Init state.

After lots of digging and packet capturing, I found the issue is that the default MTU setting is too big for the WireGuard tunnel. This can be lowered at the server side bird.conf and doesn't require updates to the client's bird.conf:

protocol ospf {
    area 0.0.0.0 {
        interface "wg0" {
            tx length 1300;
            type ptmp;
        };
    };
}

Change 1300 to whatever MTU the WireGuard tunnel uses, birdc configure, and everything should be working smoothly:

$ sudo birdc show ospf neighbors
BIRD 1.6.6 ready.
ospf1:
Router ID   	Pri	     State     	DTime	Interface  Router IP   
192.168.123.2	  1	Full/PtP  	00:31	wg0        192.168.123.2   
192.168.123.3	  1	Full/PtP  	00:31	wg0        192.168.123.3

OSPFv3

OSPFv3 config are exactly the same as described in the OSPF section above, except the following:

Originally all my clients used a public IPv6 subnet from the server and neighbors inside bird6.conf on the client used the public IPv6 address of the server, it will not work! You must config and specify a link-local address on the bird6.conf instead (such as fe88::1/64).

The BIRD documentation for OSPF protocol actually briefly mentions this:

In OSPFv3, link-local addresses should be used, using global ones is possible, but it is nonstandard and might be problematic. And definitely, link-local and global addresses should not be mixed.

However, during my test, using global addresses just simply doesn't work at all, despite all other configs being the same.

If the MTU of the WireGuard tunnel is small, then you must set the tx length 1300; config on the server side accordingly as well.

Here are the end results for OSPFv3:

$ sudo birdc6 show ospf neighbors
BIRD 1.6.6 ready.
ospf1:
Router ID   	Pri	     State     	DTime	Interface  Router IP   
192.168.123.2	  1	Full/PtP  	00:33	wg0        fe88::2                               
192.168.123.3	  1	Full/PtP  	00:36	wg0        fe88::3