PlugIn: YouTube (2015)

[tmiranda]

The API has hit a quota limit. Since I do not pay, limits are set on the number and type of queries that can be done over certain periods of time. The quota for some of the APIs have been exceeded due to a big spike on 2/21.

I think the error will go away when the usage stats are reset. I don't know exactly when that will be because I've never seen this error before.

[gary201]

That's interesting because the same issue is happening with the PlayOn Windows application as well. I can see what I've subscribed to, but nothing is visible.

[jerrym]

I also have the same issue, and it started mid morning on the 21st.

Is it possible the transaction count spiked because of some sort of error that sent the code into a retry?

Love the functionality and the hard work that went into developing it.

Is there any additional information can I provide to help with this issue?

[tmiranda]

Originally Posted by jerrym

I also have the same issue, and it started mid morning on the 21st.

Is it possible the transaction count spiked because of some sort of error that sent the code into a retry?

Love the functionality and the hard work that went into developing it.

Is there any additional information can I provide to help with this issue?

I don't really know why it's happening. I will look into it later today or tomorrow. I was hoping it would clear up on its own.....

[gary201]

Who owns the project for the API used? Is it yours or someone out on GitHub or somewhere? I did some searching around and there are comments on the net from the past where the quota needed to be increased. I've no idea whether that applies here or not, but it certainly seems plausible given what you reported earlier.

[tmiranda]

Originally Posted by gary201

Who owns the project for the API used? Is it yours or someone out on GitHub or somewhere? I did some searching around and there are comments on the net from the past where the quota needed to be increased. I've no idea whether that applies here or not, but it certainly seems plausible given what you reported earlier.

I'm the owner. I suspect you are correct. When I look at the daily quota it is blank. I took a first stab at increasing the quote and failed. It's a complicated form that gets submitted to Google. It seems more geared at a large project rather than the modest needs of my PlugIn. I plan on giving it another shot tonight.

I'm not sure what changed in the first place. Maybe because I haven't done anything with the project for a while?

[Beaker]

Maybe this is due to COPPA compliance attempts?

[tmiranda]

I just updated the quota limit. Let me know if it's working or not.

[jerrym]

Originally Posted by tmiranda

I just updated the quota limit. Let me know if it's working or not.

It does not appear to be. I am still getting the same "There are no new Videos." or "There are no favorite channels" for the menu items and the downloading is not working.

From the Log file:

Fri 2/28 9:41:47.345 [Timer-3@3de4e07d] YT: Subscriptions.getSubscriptionsForAuthorizedUser: JSON error. 403 : Access Not Configured. YouTube Data API has not been used in project 679168243199 before or it is disabled. Enable it by visiting https://console.developers.google.co...t=679168243199 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.

[timzool]

This was annoying me so I started doing some of my own research. I went over to Google developers and set up my own API key. You see, after some poking around I discovered that this plugin does not hard code the API Keys it uses. The API Keys are a folder named YouTube.properties in the root stageTV directory. You can create your own key, it's free with limitations of course. Most notably being the quota limit that's causing all the problems here. I was up and running in moments but I made some disturbing discoveries. I really don't think any of your Solutions are going to work. The YouTube data API has a quota limit of 10,000 queries for a free account. A routine download of my favorite channels videos amounted to over a thousand queries. I don't know if this is normal behavior for the Plug-In or if this represents a change of its interaction with the API. You really need to check on this. Also, I would like to know if this thing should be making any queries when it's not in use. Anyways, with multiple users on a shared API key I don't see how this thing could really function at all. However, until recently it was working just fine so I don't know if something changed. By the way, I'm just using an API key. I'm not bothering to use an OAuth 2.0 key because I discovered that that's only needed if you intend to log into YouTube. I never use this plug-in that way so I don't need it. I changed my download check frequency to once every 24 hours. for the next couple of days I'll keep track of how many queries SageTV makes to see if this works all right.

[Beaker]

Thanks for looking in to this! It has become my most used way of watching any video media.

For what it's worth I'm still seeing the same behavior and error in logs as jerrym.

[Jaroch76]

Originally Posted by timzool

This was annoying me so I started doing some of my own research. I went over to Google developers and set up my own API key. You see, after some poking around I discovered that this plugin does not hard code the API Keys it uses. The API Keys are a folder named YouTube.properties in the root stageTV directory. You can create your own key, it's free with limitations of course. Most notably being the quota limit that's causing all the problems here. I was up and running in moments but I made some disturbing discoveries. I really don't think any of your Solutions are going to work. The YouTube data API has a quota limit of 10,000 queries for a free account. A routine download of my favorite channels videos amounted to over a thousand queries. I don't know if this is normal behavior for the Plug-In or if this represents a change of its interaction with the API. You really need to check on this. Also, I would like to know if this thing should be making any queries when it's not in use. Anyways, with multiple users on a shared API key I don't see how this thing could really function at all. However, until recently it was working just fine so I don't know if something changed. By the way, I'm just using an API key. I'm not bothering to use an OAuth 2.0 key because I discovered that that's only needed if you intend to log into YouTube. I never use this plug-in that way so I don't need it. I changed my download check frequency to once every 24 hours. for the next couple of days I'll keep track of how many queries SageTV makes to see if this works all right.

Thank you for the research; it would never have occurred to me that the API key wasn't hard-coded.

Like many others, this plugin has become a key part of my video-watching experience. As far as I'm concerned, if this means generating my own API key, I'm fine with that.

Also, in case I forget to say so later, thank you tmiranda. Your plugin is wonderful.

[tmiranda]

Originally Posted by timzool

This was annoying me so I started doing some of my own research. I went over to Google developers and set up my own API key. You see, after some poking around I discovered that this plugin does not hard code the API Keys it uses. The API Keys are a folder named YouTube.properties in the root stageTV directory. You can create your own key, it's free with limitations of course. Most notably being the quota limit that's causing all the problems here. I was up and running in moments but I made some disturbing discoveries. I really don't think any of your Solutions are going to work. The YouTube data API has a quota limit of 10,000 queries for a free account. A routine download of my favorite channels videos amounted to over a thousand queries. I don't know if this is normal behavior for the Plug-In or if this represents a change of its interaction with the API. You really need to check on this. Also, I would like to know if this thing should be making any queries when it's not in use. Anyways, with multiple users on a shared API key I don't see how this thing could really function at all. However, until recently it was working just fine so I don't know if something changed. By the way, I'm just using an API key. I'm not bothering to use an OAuth 2.0 key because I discovered that that's only needed if you intend to log into YouTube. I never use this plug-in that way so I don't need it. I changed my download check frequency to once every 24 hours. for the next couple of days I'll keep track of how many queries SageTV makes to see if this works all right.

I knew from the beginning that a shared (free) key would not be suitable for heavy use. It did work fine for the limited use needed by the plugin. They must have changed something but I sure can't see what. My current quota is at 0 per day and I can't seem to increase it

Generating your own key as you described will work. That's the main reason I put the key in the properties file. (Clearly, this would not be a good thing to do if the plugin was a "real" project.)

[gary201]

Is that something you think we're all going to need to do (create our own keys)?

[jerrym]

I tried to create my own key and put it in the properties file, but it is not working for me. I am not getting the error message in log file anymore.

I do login to YouTube, so that might be a difference.

Could someone help with some instructions on how to properly create and configure the API key?

[timzool]

Originally Posted by jerrym

I tried to create my own key and put it in the properties file, but it is not working for me. I am not getting the error message in log file anymore.

I do login to YouTube, so that might be a difference.

Could someone help with some instructions on how to properly create and configure the API key?

It's not enough just to create a key. First you need to create a project. Then you need to add the "YouTube Data API v3" to your project. Now, I don't log into YouTube, so, I don't know what else is required for that. There are a couple more YouTube apis. All that I'm using is the "YouTube Data API v3". You'll have to ask timiranda if there is anything else required for logging in. When you create your project, it will be created with a whole bunch of apis that you don't need. From the menu go to home, then go to apis overview. On the bottom of the screen they will all be listed. You can just disable them one by one. Add the YouTube data API by clicking on apis and services above. Then you simply find it and add it. Then of course, you have to create your API key. On the menu it's under apis and services/credentials. And if you intend to log into YouTube, You'll also need the OAuth 2.0 client ID and client secret keys. Again, if this comes with a need for other apis I don't know.

[Jaroch76]

I finally sat down and created my own API key this morning, and it seems to be working. Thank you to everyone who has put hard work into building this wonderful program and debugging the latest issue.

For those who having trouble navigating the less-than-straightforward process of creating an API key, here's what I had to do:

1. In my web browser, I went to https://developers.google.com/youtub...etting-started. This gave me the basic instructions I needed for starting.
2. Following their instructions, I opened the Google Developers Console.
3. I followed their prompts to create a new project, and attached the YouTube Data API v3 API to this project.
4. I then navigated to the Credentials section of the API dashboard.
5. I created an API key and an OAuth 2.0 Client ID.
6. I navigated to the OAuth consent screen section and set up a consent screen. I don't know if this was needed; Google said I should do it.
7. I then opened the directory containing SageTV and the YouTube-DL configuration file.
8. After backing up the configuration file, I edited it, putting my new API Key, OAuth Key, and OAuth Client Secret in the respective fields.
9. Then I loaded SageTV, and logged out of YouTube.
10. I logged back into YouTube, and used the URL and Device Code provided by SageTV to tell Google that, yes, SageTV was allowed to connect to YouTube.
11. Somewhere along the lines, I got a security alert email from Google. I followed the link in this email to acknowledge the alert and make Google happy.

I may have missed something. But those are the important things.

[Jaroch76]

I created my own API key earlier this morning. For about twenty minutes, it worked.

But within that short span, the quota has already reached 9756 queries.

Now, the log file is full of lines like these:

Thu 3/5 6:20:06.282 [Timer-1@32d108dd] YT: Search.getNextPage: JSON error. 403 : Daily Limit Exceeded. The quota will be reset at midnight Pacific Time (PT). You may monitor your quota usage and adjust limits in the API Console: https://console.developers.google.co...t=258704486810

More worrisome, after a few minutes, the quota-errors went away to be replaced by these:

Thu 3/5 6:21:36.775 [Timer-1@32d108dd] YT: Search.getNextPage: JSON error. 401 : Invalid Credentials

The error messages were being written to the log file at a rate of several times per second. I eventually had to restart the SageTV service to stop the runaway spamming.

Also, during the short period that it worked, it downloaded five videos... but downloaded them to the "youtube_downloads" subdirectory in the SageTV directory, not the directory where they're configured to go.

I don't know if it'll help, but I've attached the log file for that time period.

[robogeek]

Just to add to this, in addition to using my own YouTube Data API key, oAuth key, and oAuth Client Secret, I also went into the YouTube (2015) options menu to change a few settings to prevent it from blowing through the laughably low 10000 unit quota limit that someone at YouTube actually thought was a good idea.

For full disclosure, I don't use the YouTube (2015) plugin for browsing or searching for YouTube content. I only use the Browse Favorite Channels menu item to manage my channel favorites when needed. I don't use the What's New, Recommended For You, My YouTube Channel, Best of YouTube, or any of those other menu items in the plugin. I suspect using those other menu items will quickly blow through the 10000 unit quota limit. I use the YouTube (2015) plugin strictly for recording new videos from 25 different YouTube channels (and averaging less than 4000 of the 10000 quota units per day) and have not had any trouble since making the following additional changes...

YouTube Menu Options:
- YouTube Menu Options >> Maximum Items to Fetch
I set Maximum Items to Fetch to the lowest number, 50. I believe the YouTube Data API still returns results in sets of up to 50 with a specific quota usage per 50 results. That is, a search request that returns 1 result would have the same quota usage as a search request that returns 50 results. A request that returns 51 to 100 results would have double the quota usage of a request that returns 1 to 50 results.

- YouTube Menu Options >> Recording Options >> Favorite Check Interval
I set this to 24 so it only checks for favorites once per day. Checking two or three times per day (or more) will result in two or three times (or more) the quota usage that pertains to checking for new videos on your channel favorites.

- YouTube Menu Options >> Advanced Options >> Include Keyword Favs in "What's New"
I set this option to Disabled. I'm not sure if it helps, but anything that looks like it may generate quota usage should be disabled if it is not absolutely needed.

For each of my YouTube channel favorites, these are the Favorite Options:
- Fetch Watched Videos: Disabled
- Include in "What's New": Disabled
- Record New Episodes: Enabled
- Get Episodes Since/From: 'A Week Ago' **
- Re-Record Deleted Episodes: No
- Minimum Duration: Any Duration
- Simulate (Do not record): Disabled

** I'm not sure how the plugin code is written regarding the API requests for new episodes, so it may be necessary to set this option to 'A Week Ago' on a regular basis...maybe once a month or once every few months, etc... depending on how many channel favorites are set and how often videos are released on the channel favorites. If the daily quota usage slowly creeps up over time, resetting this setting to 'A Week Ago' again may bring quota usage back down.

[gary201]

I can't help but wonder what other PVR implementations are doing for YouTube. Are they equally affected or do they have a completely different implementation so that they are unaffected by whatever change Google has done here? Or is it the case that all PVR solutions will require each user to create their own projects, keys tokens, etc.?

One interesting coincidence is that PlayOn may similarly be affected. I similarly noticed that the PlanOn Windows app can't see that anything exists for any of my subscribed YouTube channels. The folks at PlayOn insist they can't reproduce this, and I've insisted that I'm not going to give them my Google credentials for further debugging (for obvious reasons since that gives them access to email, and other things authenticated by Google). Is it possible that the YouTube plugin for Sage could be affecting the ability for PlayOn to also see that same content?