Bernardo Cardozo
04/19/2024, 2:25 PM"body": {
"call": "ListarMovimentos",
"app_key": "x",
"app_secret": "x",
"nRegPorPagina": 1,
"nPagina": 5
}
Is ther a way to get this into the "param" field inside the request body?Antonio Grass
04/19/2024, 8:09 PMAmit Bose
04/19/2024, 10:42 PMMichal Krawczyk
04/22/2024, 8:58 AMincremental_sync
yaml node should be used in a Full Refresh mode? I'd expect that when the Full Refresh mode is used the incremental_sync
yaml node is just ignored and the stream would just fetch all data using the defined pagination strategy without using incremental parameters. Aparently this seems to not to be the case and from what I observed the Full Refresh mode just doesn't record the final stream state. That said, is there a way to implement an incremental stream (let's say with updated_at
cursor), but initially when a stream is created run a full refresh of all objects regardless when they were last updated?Christian Schmid
04/23/2024, 10:39 AMMateo James
04/24/2024, 10:19 AMAmit Bose
04/24/2024, 5:01 PMJustin Lemmon
04/25/2024, 4:16 AMairbyte-ci connectors --name=source-quickbooks build
to test some connector updates and I'm getting the error:
failed to start daemon: Error initializing network controller: error obtaining controller instance: unable to add return rule in DOCKER-ISOLATION-STAGE-1 chain: (iptables failed: iptables --wait -A DOCKER-ISOLATION-STAGE-1 -j RETURN: iptables v1.8.10 (nf_tables): RULE_APPEND failed (No such file or directory): rule in chain DOCKER-ISOLATION-STAGE-1
Any thoughts? I've already tried the suggestions of sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
that I've seen in a few threads. Running Airbyte on Docker via WSL2 + Ubuntu.Yeongbae Jeon
04/25/2024, 6:17 AMYeongbae Jeon
04/25/2024, 6:25 AMMarcos Marx (Airbyte)
04/25/2024, 2:01 PMRenan Soares
04/25/2024, 2:05 PMMarcos Marx (Airbyte)
04/25/2024, 3:47 PMJohn Mikkelson
04/25/2024, 3:54 PMairbyte-ci connectors --name=source-facebook-marketing build
, but I'm running into issues with the Dagger dependency. It seems that there is some kind of error when trying to install the correct Dagger version. I also tried to install Dagger=0.9.6 manually, without luck (it defaults to latest and cannot be changed). Here is the error message:
WARNING pipelines.cli.dagger_run: The Dagger CLI version '0.11.1' does not match the expected version '0.9.6'. Installing Dagger CLI '0.9.6'... dagger_run.py:92
/bin/sh: symbol lookup error: /bin/sh: undefined symbol: rl_trim_arg_from_keyseq
Traceback (most recent call last):
File "airbyte_ci.py", line 210, in <module>
File "asyncclick/core.py", line 1205, in __call__
File "anyio/_core/_eventloop.py", line 68, in run
File "anyio/_backends/_asyncio.py", line 204, in run
File "asyncio/runners.py", line 44, in run
File "asyncio/base_events.py", line 649, in run_until_complete
File "anyio/_backends/_asyncio.py", line 199, in wrapper
File "asyncclick/core.py", line 1208, in _main
File "asyncclick/core.py", line 1120, in main
File "asyncclick/core.py", line 1736, in invoke
File "asyncclick/core.py", line 1485, in invoke
File "asyncclick/core.py", line 824, in invoke
File "click_decorators.py", line 89, in wrapper
return await f(*args, **kwargs)
File "click_decorators.py", line 89, in wrapper
return await f(*args, **kwargs)
File "click_decorators.py", line 89, in wrapper
return await f(*args, **kwargs)
File "airbyte_ci.py", line 197, in airbyte_ci
File "dagger_run.py", line 115, in call_current_command_with_dagger_run
dagger_path = check_dagger_cli_install()
File "dagger_run.py", line 95, in check_dagger_cli_install
install_dagger_cli(expected_dagger_cli_version)
File "dagger_run.py", line 57, in install_dagger_cli
subprocess.run([install_script_path], check=True)
File "subprocess.py", line 526, in run
subprocess.CalledProcessError: Command '['/tmp/install_dagger.sh']' returned non-zero exit status 127.
[75713] Failed to execute script 'airbyte_ci' due to unhandled exception!
Fernando Lopez
04/25/2024, 5:47 PMJerry Bao
04/26/2024, 7:25 AMairbyte-ci
or is there another way? Is there a Dockerfile sample I can reference?Rocky Appiah
04/26/2024, 2:12 PMdatetime
to perform incremental syncs, but rather cursor paged pagination to continue from where you left off.
Looking at the docs for cursor paged pagination, it doesn’t appear to store this value for future syncs. Is there a way I can reference this value in incremental syncs? I see Filtering according to Cursor Field, but this appears to be a datetime
value, no a string.Ethan Steininger
04/26/2024, 5:06 PMRobert Evans
04/26/2024, 5:23 PMBrian Henson
04/27/2024, 7:10 PM/company/{{ config['realm_id'] }}/query?query=SELECT * FROM Account STARTPOSITION {{ next_page_token }} MAXRESULTS 2
so the {{ next_page_token }}
variable is driven by a fixe offset pagination, but my smooth-brain can't figure it out from the documentation. I'm feel THIS CLOSE, but need help to get it across the finish line. Any guidance would be GREATELY appreciated!Alexandre Girard (Airbyte)
04/29/2024, 7:54 PMlast_records
interpolation variable that can be pointed to from the CursorPagination component is being deprecated and replaced by
- `last_record`: pointer to the last record read
- `last_page_size`: Size of the last page of records
These two new variables can be used to replace your usage of last_records
without needing to keep all records in memory.
Ask:
Please update references to last_records. Most changes should look like
- {{ last_records[-1] }}
-> {{ last_record }}
- {{ last_records | last }}
-> {{ last_record }}
- {{ last_records.size() }}
-> {{ last_page_size }}
- {{ last_records|length }}
-> {{ last_page_size }}
- {{ if last_records }}
-> {{ if last_record }}
The last_record
and last_page_size
variables were already introduced in 0.78.4
. Support for last_records
will be removed from the CDK within in the next 2-3 weeks.
We'll help Cloud users migrate, but you'll need to manually update your connectors if using the self-hosted Airbyte platform.
Let me know if you have questions!Vivek Jain
04/30/2024, 5:27 AMTask airbyte integrationsconnectors:checkPythonUsing python 3.11.5 from /root/airbyte/airbyte/airbyte-integrations/connectors/build/.venv (/root/airbyte/airbyte/airbyte-integrations/connectors/build/.venv/bin/python) Using pip 21.0.1 from /root/airbyte/airbyte/airbyte-integrations/connectors/build/.venv/lib/python3.11/site-packages/pip (python 3.11)
Task airbyte integrationsconnectorsdestination verticacompileJava FAILEDFAILURE: Build failed with an exception. * What went wrong: Execution failed for task 'airbyte integrationsconnectorsdestination verticacompileJava'.
error: invalid source release: 21* Try:
Run with --stacktrace option to get the stack trace.
Run with --info or --debug option to get more log output.
Get more help at https://help.gradle.org.Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0. You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins. For more on this, please refer to https://docs.gradle.org/8.6/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation. BUILD FAILED in 33s 11 actionable tasks: 3 executed, 8 up-to-date Any thoughts on this one ? Regards/Vivek
Yepher
04/30/2024, 4:29 PMLake (Airbyte)
04/30/2024, 10:04 PMIncremental Sync
is configured, the state
output by each partition can now be viewed in the testing panel, and users can set the Input state
to use for the next test read.
◦ This allows connector developers to verify that the incremental state matches the expected shape and is compatible with existing state, and to test how the connector responds to different input states.
◦ In the output state tab, there is also a Use as Input State
button which allows users to easily use the output state as the input state for the next test read with a single click
◦ See first 2 screenshots below for an example of how this looks in the UI
• The Authentication
component can now be switched to YAML-mode, allowing users to use custom authenticators (by following this guide) or authenticators that are not yet added to the UI
◦ This includes another subtle feature: when switching to YAML mode, any auto-created user inputs will remain and be "unlocked" so that users can modify them as they wish
◦ When switching the component back to UI mode, the required auto-created inputs are verified to have the required properties, and are re-locked upon switching to UI
◦ See the last screenshot below for an example of how this looks in the UIMert Karabulut
05/01/2024, 8:33 AMMadiha Khalid
05/01/2024, 11:35 AMKeegan Haukaas
05/01/2024, 6:44 PMRobin Herzog
05/01/2024, 7:11 PMAchin Kumar
05/02/2024, 5:27 PMfaros-airbyte-cdk
with OAUTH authentication where a refresh token becomes invalid after it has been used to create an access token. In these situations, a new refresh token is returned along with the access token.
How do i go about updating the new access and refresh token in the source?Aaron Lim
05/03/2024, 12:51 AM