Log in
Feedback and Planning

Feedback and Planning

Discussions about the past, present, and future of Elgg and this community site.

Entity List spec

  • Public

Entity List spec

Started by Steve Clay 846 days ago Replies (48)

It's my understanding that something like an entity list will eventually make it into core, but I have an immediate need for this and must push forward coming up with an API that I can implement now in a plugin. I think it'd be mutually beneficial to make my implementation as close to the final one as possible, so I welcome input here.

My main idea is that an ElggList should be a lightweight tool rather than being a full blown ElggObject. Two new tables:

  • elgg_lists: list_id (AUTOINC), key_sha1 (char 40, default NULL)
  • elgg_list_items: list_id, item_id (int, NOT NULL), weight (int, default NULL)

API usage:

$list = new ElggList(); // list with NULL alias


$list = new ElggList("user:123|stickyPages");

$list->id // (read only)

$list->setItems($arrayOfGuids); // most efficient way to establish a list

$list[] = $guid; // adds an item row

array_shift($list); // remove first row

You could always reference a list by ID (available via $list->id), but you could store the SHA1 of a string in `key` like in my original proposal. If key is given, it should be unique across lists.

$list->key = "user:123|stickyPages"; // or set it in the constructor

Static methods findById($id) and findByKey($key) that would return null if not found.

If someone wanted a list to have a name/description, they could just make an object and hold a reference to the list's ID in metadata:

$photoAlbum->list_id = $list->id;

Then we need API methods to JOIN the elgg_list_items table inside get_entities in various ways. E.g.:

$listEntities = elgg_get_entities_in_list($list->id);
$markup = elgg_list_entities_in_list($list->id);
$entitiesSortedByList = elgg_get_entities(array(
    'sortByList' => $list->id

So... thoughts?


  • Ismayil Kharedinov 846 days ago

    Hi Steve,

    Good one. Thumbs up. I was actually starting to work on my implementation of this, but eventually gave up as it wasn't really making my life with lists easier.

    However, here are my thoughts:

    1. I like your original proposal where elgg_get_entities options are stored with the list

    2. It would be good to have an ECML compatible key

    3. I would encourage more Elggy 'keywords' - 'item_guid' instead of 'item_id', 'priority' instead of 'weight', and same goes for static methods

    4. Privileges - who can add items to the list? who can view the list?

    5. I would like to see the following methods:

    $list->addItems($arrayOfGuids) // there must be a way to specify priority

    $list->getItems($options) // options to narrow down the return

    $list->removeItems($arrayOfGuids) // there must be a way to re-build the priorities

    $list->setItemPriority($item_guid, $priority)


    $list->view() // html markup, must accept an array of $vars to be used in page/components/list


    $list->canEdit() // perhaps plugin hooks would suffice

    6. Helper API functions

    elgg_get_list_by_id() // from here you can use static methods to get list items


    elgg_view_list($list_entity, $order_by, $vars)


    7. It must be insured that items are removed from the list if the entity is deleted

    8. Should the items be added to the list automatically if they match $list->options? 

    9. It would be good to see lists of mixed type/subtype (not just object/blog, or user/all, but 2 blogs, 3 files etc.)

  • Cash 845 days ago

    1. I greatly prefer calling these collections to lists since we already use the term lists in Elgg

    2. Agree with Ismayil on using item_guid and priority rather than list_id and weight

    3. Why the hash rather than storing the name of the list?

    4. Seems lists need owners. How else do you control adding to the lists or knowing what lists a user has?

    5. Not making this an entity means no commenting on lists, no notifications based on lists, no activity stream updates on lists, and no relationships with lists without having a paired ElggObject. If we end up with a lot of lists needing the paired ElggObject for one of these reasons, we may end up with a system that is slower than just making it an ElggEntity. We would need to develop a set if use cases to evaluate this. It is easier to promote something to an entity than to demote it so the side to error on is making it not an entity.

    6. Your original proposal had a site guid in the list definition. That would allow it to support multi-site Elgg installs.

    7. If having a paired ElggObject is going to be a common use case (though not common enough to justify promotion of the list to ElggEntity), I prefer that the link to the ElggObject be built into the list rather than it being metadata.


    This is obviously something that we cannot pull into 1.8, but 1.9 seems reasonable.

  • Kevin Jardine 845 days ago

    It is not clear to me from Steve's original post why he needs this feature. Steve can you explain more?

    I would especially be concerned about adding new tables as I would argue that the Elgg table structure is already more complex than necessary and a clean and simple data storage system has many advantages.

  • Kevin Jardine 845 days ago

    It would also be good to understand why a list cannot simply be created using Elgg's current container system. Why not create an object with subtype list?

  • Kevin Jardine 845 days ago

    I just noticed Steve's original proposal link. Sorry I had not noticed that before.

    It is common to create a container for a collection of objects and then attach a display_order field to each object.

    Is your point that you want to have the same item in multiple lists with a different display_order for each list?

    In that case, why not just serialize a list of guids? Is that because it violates the rule of not including guids in metadata?

  • Cash 845 days ago

    Serializing GUIDs is clunky and makes clean-up more difficult than it needs to be. if the choice is between adding a list/collection table and using serialized GUIDs, adding the table definitely wins. The fact that the only way to accomplish this right now is with serialized GUIDs means Elgg is lacking something (and yeah, I want to be able put the same entity in multiple collections).

  • Kevin Jardine 845 days ago

    Cash, there is nothing wrong in my view if Elgg is "lacking something".


    One of Elgg's major advantages is the clarity and simplicity of its API. The more features that are added, the more code there is to maintain, the more opportunities for bugs, the steeper the learning curve, the fewer the developers there will be and the less popular the application.

    I am not saying that there isn't a case to be made here for a new feature but I think that it needs to be stronger than that Elgg is "lacking something".

  • Kevin Jardine 845 days ago

    I think that a case could be made for storing a guid ordering to avoid having to serialize one.

    But why create a special collection entity?

    Would not a single table:

    container_guid, item_guid, priority

    do the job?

    Any container could optionally have an ordering associated with it.

    Items within an ordered collection would contain a reference to a container as usual but developers could optionally store an ordering in an ordering table as well.

    So instead of adding the idea of an ElggList or an ElggOrderedCollection we would have the lighter weight idea of an ElggOrdering.

  • Kevin Jardine 845 days ago

    One advantage of the ElggOrdering approach is backwards compatibility. If no ordering was available, Elgg could default to the default behaviour (eg. guid descending).

    EDIT: But now that I think more about it, the container approach would not work because it is hierarchical. As I mention below, this would need to work with relationships (many-to-many) instead.

  • Cash 845 days ago

    The model that Brett and I have used for adding functionality to Elgg is that Elgg should provide 80% of the functionality for 80% of the developers. This still leaves us with a subjective decision on what should be included and what shouldn't, but it does provide structure for the discussion.

    We've been doing some Milestone planning (with the objective of getting input from the wider community soon). Adding collection support was one of the items listed in the near term so it passed the above threshold for the three of us.

    The reason I think Elgg needs this is because too many of us are implementing custom collections/ordered lists. We have two custom ordered lists in Elgg core: widgets and plugins. We have had requests to add ordering to pages in the pages plugin. If the sites pages plugin provided the ability to create arbitrary pages, we'd want those to also support arbitrary orderings. Outside of core/bundled plugins, the Tidypics plugin has an implementation of custom ordering. Steve provided a few use cases in his original proposal and I could name several others.

  • Kevin Jardine 845 days ago

    Hi Cash,

    In the case of clear hierarchies (eg. plugins in a site, photos in an album, a page hierarchy) would not the standard display_order option suffice?

    I agree that widgets are a different case because a given widget can appear on multiple pages so having the ability to specify a different ordering would help.

  • Kevin Jardine 845 days ago

    And, of course the hierarchical relationship is exactly why the current container system would not work with Steve's use case.

    We would need the ability to order relationships, not containers.

  • Kevin Jardine 845 days ago

    To add one more thought to this:

    Currently the standard way to allow multiple entities to "contain" the same items is through relationships.

    For example, users can have the "member_of" relationship with multiple groups.

    Relationships now have timestamps but that is the only way to sort group members. To implement Steve's use case, we would need to have other ways to order relationships associated with an entity.

    Does this make sense?

  • Kevin Jardine 845 days ago

    And if it does make sense, then perhaps an extra field added to my proposed single table would do the job:

    relationship, guid1, guid2, priority

    (Edited for clarity.)

  • Kevin Jardine 845 days ago

    Ahh - would not adding a single priority field to the relationships table do the job as well?

  • Ismayil Kharedinov 845 days ago

    @Kevin, Since 1.8, I've been trying to create hierarchical systems via containers, and it's a real pain. Over time I've realized there is a difference between a containing entity (a way to describe a relationship, and build an hierarchy logic) and a containing element (look at it as a DOM element where we want to the entities to show). For example, a widget is not a container per se, rather a placeholder for a collection of entities. Having lists/collection would resolve the problem of excessively adding metadata, e.g. in case when I have an hierarchy of nested placeholders - let's say I have an entity that has a logical division into segments/categories, which further have segments/categories inside, which then contain widgets. The amount of metadata/relationships I have to use to make this work is scary. Having a collection would eliminate that.

    However, I do agree that perhaps Relationships would be a good way to go forward. But then again you would still need to have a collection as an entity that has a guid to establish a relationship.

    @Steve, one more thing to add to the list is the position of the entity in the list. Similar to widgets, you would want to have an ability to control in which column/section/category an entity appears in the rendered list (in case it's rendered e.g. as a table)


  • Kevin Jardine 845 days ago

    Ismayil, currently any Elgg entity can have any relationship with any other entity. I see no reason to change that.

    All we would be doing would be adding a way of ordering the relationships between one entity and the entities it has a given relationship with.

    I think that would deal with Steve's use case.

    In the case of current containers, we would simply make it clear that Elgg's containers are  one-to-many *and* unordered. If you want something else, use relationships. We can then extend the elgg_get_entities_from_relationship API to support the new ordering feature.

  • Ismayil Kharedinov 845 days ago

    My point was that to establish a relationship with a list you would need to create a collection object first. You might as well have a class that defines methods for this subtype.

  • Steve Clay 845 days ago

    @Ismayil The nice thing about a collection being just a joined table is that a single set of rows can not only order a flat list, but also a hierarchical list, too: you just JOIN the table in with each branch fetch. The priority acts kind of like XPath document order.

    @Kevin My first use case will be each user having a collection of their favorite entities (mainly publication objects synced from Drupal) that they could re-order. Eventually I want to make a topbar menu item "★" that opens a dropdown list of the first 10 or favorites links.

    @Cash/Ismayil I'm with you on collections needing access control/ownership/metadata/name/description, so ElggCollection should probably extend ElggObject.

    My attraction to the "key" idea was to make finding collections one quick query with no joins, but this would also bypass access control, so I'm OK with letting it go. Using a collection will require two access-controlled queries: one to pull the metadata/relationship to find the collection GUID, and another allowing the user to get a reference to the collection.

    Which of (metadata, relationship) is the more appropriate way to link a collection to an entity?

  • Ismayil Kharedinov 845 days ago

    @Steve, why not annotate entities? I think all the API you need is already in place. User 'favorite' as annotation name and priority as an annotation value?

  • Cash 845 days ago

    The amount of joins needed to use annotations for sorting the favorite list is prohibitive if it's going to be a common function of a site.

  • Kevin Jardine 845 days ago

    Ismayil, in Elgg all entities can be containers and indeed the notion of container is really just a faster way to implement the relationship "hasContainer". That has a simplicity and elegance that I think would be lost if we introduced a special type of entity for that.

    Steve, I think that ordering relationships would give you just what you want. No further machinery should be necessary so far as I can see. You could have a relationship "hasFavourite" which would be defined between the user entity and any other Elgg entity.

  • Steve Clay 845 days ago

    @Kevin I definitely see relationships with priority supporting the same use cases, and being an attractive way to get this implemented more simply, but I don't like it for a few reasons:

    1. Each collection item would require a full relationship: rows in two tables, the increment of GUID, and a bunch of duplicated collection/access/owner/relationship type data. This would balloon the size of the entities/relationships tables, making essential queries slower. With a dedicated table, each collection item would only be a row of 3 int columns and joining that table would be quick.
    2. LEFT JOINing the collection_items tables allows short collections to push its members to the top/bottom of results without filtering out items. Getting that functionality with relationships would be messy.
    3. Conceptually I think of a collection as a preference of a particular user, whereas most relationships don't have an inate ordering: I have many cousins and belong to many groups but there's no order to them, but I might want to order the display of groups on my profile page. 

    Here's what I think it'll be like to use the API I now have in mind:

    $collection = elgg_create_collection($user->guid, 'stickies');
    // a collection object is created as well as a relationship of type "stickies" connecting the two
    // adding a name/description is optional

    // items are stored immediately


    $collection = elgg_get_collection($group->guid, 'stickies');
    // as long as you can access the relationship and the collection (and it exists), you get an ElggCollection.

    $entities = elgg_get_entities_in_collection($collection, $get_entities_options);
    // function above requires 1st arg to be an ElggCollection or null

    or if you want ordering but not filtering:

    $entities = elgg_get_entities_ordered_by_collection($collection, $get_entities_options);

    Is this looking better?

  • Ismayil Kharedinov 845 days ago

    $entities = elgg_get_entities_ordered_by_collection($collection, $get_entities_options);

    Not sure I understand this one... Do you mean get entities in a collection by priority?

  • Kevin Jardine 845 days ago

    Hi Steve,

    I'm suggesting adding one priority field to the relationships table.

    I don't understand how that would balloon anything?

    So far as I can see this would add much less data than two new tables.

    I'd argue that ordering relationships would be a much more natural extension to the Elgg API then to introduce a new list concept.

You must log in to post replies.