Creating custom ribbon commands is all inline with the SharePoint User Experience, and thus something you should keep in mind whenever you are developing for SharePoint 2010. However I found myself in need of creating a new group within an existing tab and stumbled upon the next MSDN article, however a minor detail is that you cannot add a CommandUIHanlder without a CommandAction. So in order to get the script working you have to delete the line with Command="EnableCustomGroup" and the line with <CommandUIHandler Command="EnableCustomGroup" />.
Leaving you with a script like you can find below to create a custom group:
1: <?xml version="1.0" encoding="utf-8"?>
2: <Elements xmlns="http://schemas.microsoft.com/sharepoint/">
3: <CustomAction
4: Id="Ribbon.WikiPageTab.CustomGroupAndControls"
5: Location="CommandUI.Ribbon"
6: RegistrationId="100"
7: RegistrationType="List">
8: <CommandUIExtension>
9: <CommandUIDefinitions>
10: <CommandUIDefinition
11: Location="Ribbon.WikiPageTab.Groups._children">
12: <Group
13: Id="Ribbon.WikiPageTab.CustomGroup"
14: Sequence="55"
15: Description="Custom Group"
16: Title="Custom Group"
17: Command="EnableCustomGroup"
18: Template="Ribbon.Templates.Flexible2">
19: <Controls Id="Ribbon.WikiPageTab.CustomGroup.Controls">
20: <Button
21: Id="Ribbon.WikiPageTab.CustomGroup.Controls.CustomButton1"
22: Command="CustomButtonCommand1"
23: Image16by16="/_layouts/images/FILMSTRP.GIF"
24: Image32by32="/_layouts/images/PPEOPLE.GIF"
25: LabelText=""
26: TemplateAlias="o2"
27: Sequence="15" />
28: </Controls>
29: </Group>
30: </CommandUIDefinition>
31: </CommandUIDefinitions>
32: <CommandUIHandlers>
33: <CommandUIHandler Command="EnableCustomGroup" />
34: <CommandUIHandler Command="CustomButtonCommand1"
35: CommandAction="javascript:alert('Hello, world!');" />
36: </CommandUIHandlers>
37: </CommandUIExtension>
38: </CustomAction>
39: </Elements>