I had all their date, times set the same.
Have the photos been taken in a different Time Zone? And did you import them at home, in a different timezone from where you took them? Have all photos been tagged with GPS before you imported them to photos, or do some photos not have GPS tags? Photos may be using wrong timezones to interpret the date and time for some photos, and then they will be sorted incorrectly, when you sort them by the time.
See: How Photos Handles Time Zones when Importing
For example, you take a picture at 9am in San Francisco and import to Photo after returning to New York. If the photo is tagged with GPS, Photos will treat the 9am a Pacific Time, using the time zone of San Francisco, but if there is no GPS tag, Photos will read the 9am Eastern Time, using the current timezone of system time on your Mac when you import the photos.
Check the timezones assigned by Photos for the photos that are sorted incorrectly. I would create a smart album for each camera model ans split the sets of photos by cameras, then then separate the smat albums further by "Photo is tagged with GPS" or not.
You can see the timezone for the date and time, if you select a photo and open the panel "Image > Adjust Date & Time". Look for the timezone strip and the "Closest City". Are there photos with a wrong timezone?